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Overview 


Policymakers increasingly recognize that early childhood programs can provide a pathway to later 
school success for disadvantaged children. However, to be effective, preschool programs must be of 
high enough quality to promote children’s development. This report presents the final results of the 
Foundations of Learning (FOL) demonstration, which evaluated an intervention designed to train pre- 
school teachers so that they could better manage children’s behavior and promote a more positive class- 
room learning environment. It was hypothesized that these improved skills could strengthen children’s 
social and emotional competence, allowing more time to be spent on classroom teaching and learning. 

FOL was tested in Newark, New Jersey, and Chicago, Illinois, using teacher framing combined with 
weekly in-class support from a master’s-level clinician to reinforce the classroom management skills 
that were covered in the training. A total of 71 preschool centers (with 91 participating classrooms) 
were randomly assigned to implement FOL or conduct preschool as usual. Differences in classroom 
practices and child outcomes between the two groups were analyzed at the end of the intervention year 
to assess the added value of FOL over and above standard preschool practice. 

Key Findings 

The evidence shows that investments in teachers’ professional development improve children’s pre- 
school experiences, although the long-term effects on children remain uncertain. 

• FOL was delivered as intended in both sites. Lower levels of institutional resources in Chicago, 
however, may have posed challenges to fully implementing the classroom-based strategies. 

• The intervention improved teachers’ positive classroom management in areas that it targeted direct- 
ly. There was some evidence in Newark that intervention classrooms had greater amounts of in- 
structional time and that, among teachers in the intervention group, positive classroom management 
was sustained one year after the intervention ended. 

• Problem behavior was reduced in the intervention classrooms. Also, FOL further improved chil- 
dren’s social and emotional competence as measured by improvements in children’s approaches to 
learning and executive function skills (attention, inhibitory control, and short-term memory skills). 

• The study provided no clear evidence that FOL improved children’s early literacy and mathematics 
skill s or that the effects described above continued beyond the preschool year, although resources 
limited the type and amount of follow-up that was possible. 

• At approximately $1,750 per child, the FOL intervention represented a 14 percent increase in pro- 
gram costs in Newark and a 21 percent increase in Chicago. Based on the limited data available in 
this demonstration beyond the preschool year, there is insufficient evidence at this point to demon- 
strate that the benefits of FOL in children’s academic gains, grade progression, or special needs des- 
ignations outweigh its cost. 
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Preface 


Federal and state expenditures for preschool programs serving children from birth to age 5 totaled 
more than $30 billion in 2011. The potential return on investment is large: high-quality preschool 
programs have been found to return as much as $4 to $10 in future benefits per dollar spent in pre- 
venting later risky behavior and boosting academic and labor market success. However, not all pre- 
school programs yield positive benefits. One possible explanation is the limited amount of time de- 
voted to “learning activities” on any given day, in part because preschool teachers are not equipped 
with the tools and techniques to effectively redirect children when their social and emotional behav- 
iors disiupt classrooms. 

The Foundations of Learning (FOL) demonstration tested one promising approach: providing inten- 
sive professional development, including in-classroom consultation by trained practitioners, to im- 
prove teachers’ classroom management skills and the social and emotional competence of young 
children who display challenging behaviors. FOL sought impacts at both the classroom level (antici- 
pating that teachers would be able to devote increased time to productive instruction) and for the 
children themselves (since enhancing their social and emotional competence was thought to be a 
strong predictor of later success in school). 

This final report has some grounds for encouragement, while also pointing to the need for further 
refinement in both the theoretical and practical issues related to strengthening early childhood pro- 
grams. While there were some promising outcomes in the FOL classrooms — more positive teach- 
er-student interactions, more effective management of challenging behaviors, lower levels of prob- 
lem behavior, higher engagement in the tasks of learning, and more instruction time — there was no 
clear evidence that FOL improved children’s short-term academic achievement, suggesting that the 
FOL approach could be part of a broader preschool strategy. 

The ongoing nationwide Head Start CARES study by MDRC was designed to build on FOL. It is 
examining three program models operating at scale and designed to enhance children’s social and 
emotional competence. The three models use distinct approaches to changing children’s social- 
emotional development, ranging from training on teacher delivery of classroom management proce- 
dures to play-based activities designed to support children’s self-regulation. Yet another effort to 
improve the return on investment of preschool programs involves the role of early mathematical 
concepts like size, shape, and counting in children’s development. Making Pre-K Count is looking at 
the effects on children’s skills of a strong developmentally appropriate preschool mathematics cur- 
riculum combined with intensive teacher professional development in math instruction. The body of 
evidence produced by these and other studies, along with the work of colleagues in the field, will 
provide critical lessons on how best to structure high-quality early childhood programs. 

Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Investments in early childhood programs are widely viewed as a promising strategy to improve 
the future educational achievement of disadvantaged young children. However, it can be 
difficult for teachers to maintain program quality if children in the classroom display challeng- 
ing behaviors. For example, when some children act out aggressively or become sad and 
withdrawn, teachers may be diverted from instructional time to manage these behaviors. Not 
surprisingly, these diversions have a ripple effect on the entire classroom of children. 

This report presents the final implementation, impact, and cost findings from the Foun- 
dations of Learning (FOL) demonstration. The FOL demonstration was designed to increase 
productive classroom time by using intensive professional development, including in-classroom 
consultation with trained practitioners, to improve teachers’ classroom management skills and 
the social and emotional competence of children who exhibit challenging behavior . 1 The 
demonstration was conducted in two cities — Newark, New Jersey, and Chicago, Illinois — 
with very different preschool systems. The Newark preschool system was particularly well 
resourced, while the level of resources in the Chicago preschool system was much lower and 
more typical of urban districts around the country. 

The report concludes that the FOL intervention was delivered with relatively high levels 
of dosage (amount of services) and quality (as rated by teachers) in both cities, with some 
differences in the focus of in-classroom consultation between Newark and Chicago. The 
program had a positive impact on teachers’ classroom management, increased instructional 
time, and improved children’s social and emotional competence during the year of the interven- 
tion’s implementation. However, there was no clear evidence of improvements in children’s 
short-term academic achievement, despite the expectation that the benefits to the classrooms 
and to children’s social and emotional outcomes would translate into better academic skills. 
Moreover, the limited data that were collected showed no evidence of sustained benefits when 
the children made the transition to kindergarten and first grade. 


Evolution of the Foundations of Learning Demonstration 

The FOL program model was initially developed to test a strong theory that had emerged from 
developmental psychology — that children’s social and emotional competence may be important 


'As used in this report, the term “competence” refers to children’s capacity to do something, and the term 
“behavior” is the manifestation of that competence in everyday life. 
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as an outcome in its own right as well as a pathway to improved academic achievement . 2 While 
FOL was being conducted, some researchers questioned this assertion, arguing convincingly that 
for academic achievement in particular, preschool academic skills may be much stronger 
predictors of later academic achievement than social and emotional competence . 3 Even so, that 
assertion is best tested in the context of studies like FOL that provide evidence about whether 
targeting children’s social and emotional competence — by, for instance, improving teachers’ 
skills in classroom management — can produce change in children’s school achievement. 

The FOL program model had previously been evaluated in CSRP (formerly known as 
the Chicago School Readiness Project) in Chicago. Encouraged by CSRP’s positive early 
results , 4 MDRC, a nonprofit, nonpartisan education and social policy research organization, 
with support from a number of foundations and the active involvement of the local school and 
Head Start programs , 5 set out to test the model on a larger scale in Newark and Chicago. 

Both CSRP and FOL sought to arm teachers with specific management strategies to 
handle daily classroom challenges, improve their interactions with children, and thereby 
enhance children’s social and emotional competence. The program model included four 
components to be delivered across the preschool year: 

• Teacher training: Lead and assistant teachers were invited to attend five 
Saturday training workshops, which were adapted from The Incredible Years 
curriculum developed by Dr. Carolyn Webster-Stratton . 6 The workshops 
provided instruction on how to foster positive relationships with children; 
they presented classroom strategies that teachers could use, such as setting 
clear rules; and they provided teachers with techniques to help children de- 
velop their social skills, anger management, and problem-solving ability. 

These strategies are widely accepted as best practices for preschool teachers 
and reflect good classroom management. 

2 C. Cybele Raver, “Emotions Matter: Making the Case for the Role of Young Children’s Emotional De- 
velopment for Early School Readiness,” Social Policy Reports 16, 3 (2002): 3-18. 

3 Greg J. Duncan, Amy Claessens, Aletha C. Huston, Linda S. Pagani, Mimi Engel, Holly Sexton, Chantelle 
J. Dowsett, Katherine Magnuson, Pamela Klebanov, Leon Feinstein, Jeanne Brooks-Gunn, Kathryn Duckworth, 
Crista Japel, “School Readiness and Later Achievement,” Developmental Psychology > 43, 6 (2007): 1428-1446. 

4 C. Cybele Raver, Stephanie M. Jones, Christine P. Li-Grining, Molly Metzger, Kina Smallwood, and 
Latriese Sardin, “Improving Preschool Classroom Processes: Preliminary Findings from a Randomized Trial 
Implemented in Head Start Settings,” Early Childhood Research Quarterly 63, 3 (2008): 253-355. 

5 Support for the Foundations of Learning demonstration came from The George Gund Foundation, The 
Grable Foundation, The John D. and Catherine T. MacArthur Foundation, The Joyce Foundation, The Kresge 
Foundation, McCormick Foundation, The Nicholson Foundation, The Pew Charitable Trusts, and the Robert 
Wood Johnson Foundation. 

6 Carolyn Webster-Stratton, How to Promote Children ’s Social and Emotional Competence (Thousand 
Oaks, CA: Paul Chapman Publishing, 1999). 
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• Classroom-level consultation: To complement the training, teachers were 
assigned a master’s-level Clinical Classroom Consultant (CCC) to work with 
them in the classroom one day per week throughout the school year. The 
CCCs built collaborative relationships with the teachers to help model and re- 
inforce effective delivery of The Incredible Years approach in the classroom. 

• Stress management: In the winter, teachers participated in a 90-minute stress 
management workshop at their program sites. CCCs also helped support the 
teachers’ use of stress management skills and techniques throughout the year. 

• Individualized child-centered consultation: Beginning in the spring, the 
CCCs provided one-on-one clinical services for a small number of children 
who had not responded sufficiently to the teachers’ improved classroom man- 
agement. By design, the individualized clinical consultation was delivered on- 
ly after children had ample time to react to the new teaching strategies. 

The timing and sequence of these activities are depicted in Figure ES. 1. 


The Foundations of Learning Demonstration 
Figure ES.l 
Intervention Timeline 


Sept. 

Oct. 

Nov. 

Dec. 

Jan. 

Feb. 

Mar. 

Apr. 

May 

June 


Monthly training sessions for teachers 


CCCs enter 
classrooms 


Classroom consultation by CCCs 


Stress management 
workshop 


One-on-one services for 
children 


NOTE: CCC = Clinical Classroom Consultant. 
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The FOL program components were expected to improve interactions between teachers 
and children, which in turn would improve a set of skills — collectively referred to as “social 
and emotional competence” — that enable children to engage positively with peers and teachers 
and to focus their attention and behavior during classroom activities, as described in Box ES.l. 
In addition, the CCCs’ one-on-one consultations with selected higher-risk children were thought 
to further reduce difficult behavior of children who needed more attention beyond the larger 
classroom management strategies. 


The Research Design 

This report draws on an analysis of program impacts in the CSRP and both FOL studies, a 
quantitative and qualitative analysis of program implementation in FOL-Newark and FOL- 
Chicago, and an analysis of program costs in FOL-Newark and FOL-Chicago. The multiple 
studies provide an important opportunity to draw inferences about how the local context might 
affect the implementation, impact, and cost of the same program model in two quite different 
urban preschool systems. 

In studying the impact of the model, all three studies (CSRP, FOL-Newark, and FOL- 
Chicago) used a rigorous research design, in which program sites were randomly assigned to 
one of two research groups: half (the program group) received the multicomponent CSRP or 
FOL intervention; the other half served as a control group that operated its regular preschool 


Box ES.l 

What Is Social and Emotional Competence? 

In this report, the term “competence” is used to indicate a child’s capacity to do something 
and “behavior” is the manifestation of that competency in everyday life. Social and 
emotional competence comprises a smaller set of discrete skills, such as the capacity to 
control negative emotions, express emotions, and communicate with peers. Social and 
emotional competence is thought to underlie children’s behaviors, particularly in two 
areas that are thought to be central to longer-temi success in school: (1) social behaviors, 
or children’s positive interactions with peers and teachers; and (2) approaches to learning, 
or children’s ability to focus their attention and behavior during classroom activities. 

Strongly related to social and emotional competence is executive function. Its underlying 
skill s are the ability to shift attention, inhibitory control (children’s ability to control their 
immediate or automatic response in favor of a planned response), and working (or short- 
term) memory. Executive function skills are thought to underlie children’s approaches to 
learning as well as their academic achievement. 
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program without the special enhancements . 7 With random assignment, differences in outcomes 
observed during the follow-up period can be confidently attributed to the program rather than to 
preexisting differences between the two research groups. Such differences, or impacts, that are 
statistically significant are unlikely to have arisen by chance. 

The study of implementation used a mixed-methods approach in which measuring the 
dosage (or amount of the intervention delivered) was complemented by qualitative data, 
observations, and surveys that shed light on the quality and process of implementation. Inter- 
vention implementation was defined as the amount and quality of training and consultation 
services that teachers received, which is consistent with The Incredible Years approach to 
intervention fidelity. The study of impacts relied on somewhat different data across the three 
studies because of funding constraints: FOL-Newark focused on trained observers’ assessments 
of the classroom environment and how children behaved in the classroom context; FOL- 
Chicago conducted standardized assessments with individual children (on their social-emotional 
and cognitive outcomes) — assessments that the literature suggests are linked to later outcomes 
in middle childhood and adolescence. Finally, a net cost analysis captured the incremental costs 
of the new services provided in the FOL intervention in Newark and Chicago, over and above 
existing services that were available to the control sites. 


Findings on Program Implementation 

• The teacher training, in-class consultation, and stress-management 
workshops were delivered at relatively high levels of dosage and quality 
(as rated by teachers) in both Newark and Chicago. However, the focus 
of in-class consultation appeared to differ somewhat between Newark 
and Chicago. 

The FOL studies demonstrate that the components of the intervention can be delivered 
in diverse implementation settings and that teachers in both settings considered FOL to be a 
worthwhile and high-quality program. Attendance was strong at the teacher training sessions in 
both Newark and Chicago, and teachers in both cities expressed high levels of satisfaction with 
the training. 

While the overall amount of in-classroom consultation was similar in Newark and Chi- 
cago, the focus of the consultation differed somewhat in the two cities. The CCCs in Newark 
engaged teachers in implementation of The Incredible Years strategies more consistently across 


7 In CSRP, the model included a classroom aide in the control group for the same amount of time as the 
CCCs in the program group. This was done in an effort to distinguish between the effect of an additional 
classroom staff person and the consultation services. 
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classrooms than was the case in Chicago, where some CCCs focused more on assisting teachers 
with basic classroom tasks (such as helping children during lunch time, monitoring recess, and 
cleaning in the classroom) than on consultation directly related to implementation of the 
classroom-management strategies. 

• Low levels of institutional resources and supports may present challeng- 
es to implementing classroom-based strategies. 

The differences in the way the CCCs spent their time in the Newark and Chicago class- 
rooms perhaps reflected the more limited staffing in Chicago. Classroom resource and staffing 
constraints in Chicago, as well as occasional lack of support by center directors and principals in 
both cities, could sometimes be major barriers to implementation. Although FOL addressed 
teacher stress, application of new classroom management strategies might still be difficult for 
highly stressed teachers in understaffed classrooms. 


Findings on Program Impacts 

Despite some differences in implementation, a consistent story emerges with regard to program 
impacts in the Newark and Chicago sites. The impacts in CSRP, FOL-Newark, and FOL- 
Chicago are summarized in Figure ES.2, in which favorable, statistically significant impacts are 
denoted by plus signs and no effect is shown as 0; “NA” (not available) indicates that the 
construct was not measured. No unfavorable impacts were found. Having two or more plus 
signs across sites for a single construct provides greater confidence than would be the case if the 
findings appeared in only a single site. 

• The intervention improved teachers’ positive classroom management. 

In addition, there is evidence that this improvement was sustained 
one year later. 

Relatively large improvements in classroom management were observed in CSRP and 
FOL-Newark, the two sites where this outcome was measured. Based on observers’ ratings, 
teachers in the program groups had more warm and positive interactions and fewer sarcastic and 
angry interactions and were better able to prevent misbehavior than were teachers in control 
classrooms. In addition, program group teachers’ positive classroom management was sustained 
one year later in FOL-Newark, where it was studied. 

• Likely because of the improved classroom management, FOL class- 
rooms devoted more time to instruction and other learning activities. 

In FOL-Newark, where this issue was examined, fewer disruptions and more orderly 
transitions appeared to free up additional classroom instructional time, with FOL classrooms 
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The Foundations of Learning Demonstration 


Figure ES.2 

Summary of Pre-Kindergarten Findings Across Studies: CSRP, FOL-Newark (FOL-N), FOL-Chicago (FOL-C) 

Teacher and Classroom Outcomes Child Outcomes 

Directly Targeted 

Problem Behavior 

Conflict/Extemalizing Behavior 
Problems (observed) 

CSRP: + 

FOL-N: + 

FOL-C: NA 

Conflict/Extemalizing Behavior 
Problems (teacher report) 

CSRP: + 

FOL-N: 0 

FOL-C: 0 

Approaches to Learning 

Executive Functioning 
CSRP: + 

FOL-N: NA 
FOL-C: + 

Attentiveness/Behavior Control 
CSRP: + 

FOL-N: + 

FOL-C: 0 

Positive Engagement 
CSRP: 0 

FOL-N: + 

FOL-C: 0 

NOTE: A plus sign (+) indicates that there was a statistically significant impact on that outcome; a zero (0) indicates that there was no statistically significant 
impact; and “NA” indicates that the outcome was not measured in the given study. 


Directly Targeted 

Positive Classroom Management 

CSRP: + 

FOL-N: + 

FOL-C: NA 


Secondary 

Classroom Productivity and 
Amount of Instructional Time 

CSRP: NA 
FOL-N: + 

FOL-C: NA 

Quality of Instruction 

CSRP: NA 
FOL-N: 0 

FOL-C: NA 


Secondary 

Positive Social Behavior 
(observed) 

CSRP: NA 
FOL-N: 0 

FOL-C: NA 

Early Academic Skills 

CSRP: + 

FOL-N: NA 
FOL-C: 0 

Social Problem-Solving Skills 

CSRP: NA 
FOL-N: NA 
FOL-C: 0 







having an average of 10 more minutes of instruction out of a 120-minute observation 
period. If this was representative of gains achieved every school day, it would translate into 
50 minutes more instruction per week, or a week’s more instruction over a school year. 
These improvements are consistent with a central hypothesis of the demonstration: that 
children with challenging behaviors may divert teachers from providing instruction to all 
children, and improving teachers’ classroom management skills can reduce unproductive 
time in preschool classrooms. 

• Problem behaviors, such as conflict among children, were generally 
reduced in the intervention classrooms. In addition, FOL usually im- 
proved children’s approaches to learning and to executive function 
s ki lls. 

There is strong evidence that the intervention improved the two child outcomes that it 
targeted most directly: children’s problem behavior and approaches to learning, with effects of 
moderate size in magnitude. With respect to problem behavior, the intervention reduced 
children’s negative interactions (conflicts) with peers. With respect to approaches to learning, 
children were observed to be more engaged in the classroom and in tasks of learning. They also 
did better on tests of executive function, a competence that underlies approaches to learning 
through a combination of attention, inhibitory control (the ability to control an automatic 
response), and short-term memory skills. 

• There was not, however, clear evidence that the intervention improved 
early academic ski ll s. 

The reduction in problem behavior and the positive impact on approaches to learn- 
ing were not consistently accompanied by positive impacts on early academic skills (for 
example, identification of letters and the sounds letters make, or ability to demonstrate basic 
addition and subtraction skills) that have been found to support later achievement in reading 
and mathematics. 

• The limited data that were collected show no evidence of sustained 
benefits for children as they make the transition into elementary 
school. It is not known whether the initial effects on approaches to 
learning or problem behavior were sustained into the early elemen- 
tary grades in either site, since data were not collected on those 
measures specifically. 

Teacher reports and school records of children’s academic achievement show no 
impacts of FOL in kindergarten and first grade. However, rates of grade repetition and 
special education were simply too low in these early elementary school years to detect 
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impacts, which may be more likely to occur later. For example, children typically do not 
complete the process for placement into special education services until at least second or 
third grade. 8 


Findings on Program Costs 

• At approximately $1,750 per child, FOL represented a 14 percent in- 
crease in program costs over normal operating costs in Newark and a 21 
percent increase in Chicago. 

Although the average cost per classroom was higher in Chicago ($34,884) than in New- 
ark ($26,873), the larger class size in Chicago meant that the costs per child were similar in the 
two sites: $1,792 in Newark and $1,744 in Chicago. These costs are higher than they are for the 
more typical approach of assigning a mental health consultant to multiple classrooms to 
intervene on an as-needed basis with children who exhibit difficult behaviors. At the same time, 
the cost of FOL is on a par with other interventions that provide dedicated, routine consultation 
services to teachers and children. 9 

• Based on the limited data of early elementary school outcomes for chil- 
dren in the FOL studies, there is no evidence that the program’s benefits 
will outweigh its costs. 

In nonexperimental research, measures of problem behavior and approaches to 
learning have been shown to be associated with benefits in these and other outcomes in the 
later school years. However, the limited follow-up data that were collected in the FOL 
studies did not show evidence that the initial effects on these measures were sustained into 
the early elementary grades. As noted earlier, it was also too soon to know whether the 
intervention affected special education placements, although reductions in such placements 
for even a small number of children would result in major savings that outweigh the 
intervention’s cost. 


8 National Association of Special Education Teachers, “Introduction to Learning Disabilities,” LD 
Report (online publication: www.naset.org/2522.0.html); The IRIS Center, “The Rationale for RTI: Early 
Intervening and Identification of Learning Disabilities,” Star Legacy Modules (online publication: 
http://iris.peabody.vanderbilt.edu/rti01_overview/rti01_04.html). 

’Florida State University, Center for Prevention and Early Intervention Policy, Mental Health Consulta- 
tion in Child Care and Early Childhood Settings: Opportunities to Expand the System of Care for Children 
with Emotional and Behavioral Challenges in Florida (Tallahassee, FL: Florida Department of Children and 
Families, 2006). 
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Implications for Policy and Research 

The CSRP and FOL studies show that a combination of teacher training and in-classroom 
consultation can improve teacher classroom management practices. FOL also had a positive 
impact on children’s behavior and approaches to learning during the preschool year, and it led 
to increased classroom instructional time, but there was not clear evidence that FOL improved 
children’s early academic skills. These and other findings from the CSRP and FOL studies 
suggest the following implications for policy and research: 

• It is important to consider the context in which early childhood pro- 
grams operate. 

This report illustrates that the FOL intervention was delivered in Newark and Chicago 
with relatively high dosage, and teachers rated the quality of the professional development 
highly. However, in these two distinct preschool contexts, differences in the level of resources 
in Newark and Chicago appeared to affect program implementation, particularly with respect to 
the focus of in-classroom consultation. A key consideration in programs of this type is the 
teachers’ receptivity to the intervention and the CCCs’ ability to engage the teachers in imple- 
menting classroom management strategies. Future efforts will need to take into account the 
level of stress that teachers feel, the resources that are available to them, and the level of support 
they receive from principals and other administrators. 

• Intensive amounts of consultation may not be necessary to produce 
measurable changes in teacher practice. 

While multiple studies on the effectiveness of classroom consultation have demonstrat- 
ed positive effects on teacher practice and child outcomes over and above any effects realized 
from training alone , 10 research is inconclusive on what the correct amount is for classroom 
consultation models . 11 The findings from the CSRP and FOL studies suggest that the level of 
intensity (a six- to seven-hour day, one day per week) and/or the duration of consultation (a full 
academic year) in this model may not be necessary to produce change in teacher practice; 
despite the lower level of consultation activities directly related to The Incredible Years training 
content in FOL-Chicago, impacts for children there were on a par with those in FOL-Newark. 
Important topics for future research include understanding the level and type of in-class consul- 


10 Bruce Joyce and Beverly Showers, Student Achievement Through Staff Development, 3rd Edition (Alex- 
andria, VA: Association for Supervision and Curriculum Development, 2002). 

"Barbara A. Wasik, Shira Kolnik Mattera, Chrishana M. Lloyd, and Kimberly Boiler, “Intervention Dos- 
age in Early Childhood Care and Education: It’s Complicated,’’ in Tamara G. Halle, Allison J. Metz, and 
Ivelisse Martinez -Beck (eds.), Applying Implementation Science to Early Care and Education Programs and 
Systems: Exploring a New Frontier (Baltimore, MD: Brookes Publishing Co., in press). 


ES-10 



tation that are needed for relatively straightforward models such as The Incredible Years as well 
as for more complicated models. 

• Supporting children’s social and emotional competence can have posi- 
tive impacts but may need to be one part of an overall strategy to 
strengthen preschool education. 

Improved classroom management and the resulting changes in children’s behavior freed 
up more time for instruction in FOL classrooms. In addition, children in these classrooms might 
well have been ready and able to take advantage of the increased instruction, since they were 
more engaged in the tasks of learning and demonstrated greater executive function skills of 
memory, attention, and inhibitory control. It is possible that these positive results did not have a 
consistent impact on children’s learning because the teachers were not trained sufficiently in the 
curricula and instructional approaches that are needed to take advantage of any increased 
opportunity for learning. 

In conclusion, the CSRP and FOL interventions presented one approach to improving 
the quality of preschool education. Continued work in this area is needed to determine the 
combination of programmatic approaches and educational strategies that will best provide a 
high-quality preschool experience at scale and will boost long-term outcomes for low-income 
children. 
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Chapter 1 

Introduction 


Many policymakers and educators view investments in early childhood programs as a promis- 
ing strategy to improve the future educational achievement of disadvantaged young children. 
Indeed, state funding has doubled in the past decade for preschool programs, which currently 
serve more than one million children nationwide. 1 For this strategy to be successful, however, 
programs must be of consistently high quality. Preschools of lower or mediocre quality may 
actually undermine low-income children’s development. 2 

One oft-mentioned concern in reaching the desired level of quality is a teacher’s ability 
to manage children who exhibit problem behavior in preschool classrooms. The behaviors 
might be externalizing (such as aggressiveness or acting out) or internalizing (such as sadness 
and withdrawal). In either case, these behaviors may distract the individual children, their peers, 
and their teachers from more productive classroom activities. The capacity to address these 
issues by improving children’s social and emotional competence may be a key component of 
high-quality preschool. As used in this report, the term “competence” refers to children’s 
capacity to do something, and the term “behavior” is the manifestation of that competency in 
everyday life. 

Social and emotional competence is thought to underlie children’s behaviors, specifical- 
ly their ability in two critical areas that are central to success in school: (1) positive interactions 
with peers and teachers; and (2) focused attention and behavior during classroom activities. 
Therefore, developmental psychologists have theorized that improving social and emotional 
competence may be both an important outcome in its own right and a possible pathway, along 
with other factors, to improved academic achievement. 3 Unfortunately, few preschool teachers 
receive sufficient training or support to promote the social and emotional competence of 
children in their classroom. 

This report presents the final implementation and impact findings for the Foundations 
of Learning (FOL) demonstration conducted by MDRC, a nonprofit, nonpartisan education and 
social policy research organization. The demonstration included a rigorous random assigmnent 
evaluation of a curriculum that was designed to train preschool teachers to better manage 
children’s behavior and promote a more positive classroom learning environment. It was 
hypothesized that by improving teachers’ classroom management skills and strengthening 


'Torre et al. (2011). 

2 Magnuson, Ruhm, and Waldfogel (2007). 

3 Domitrovich, Cortes, and Greenberg (2007); Raver et al. (2008). 
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children’s social and emotional competence and improving their behavior, more time could be 
spent on teaching and learning in classrooms. The FOL intervention was tested in two cities — 
Newark, New Jersey, and Chicago, Illinois — using an adapted version of The Incredible Years 
curriculum supported by Clinical Classroom Consultants (CCCs) who modeled and reinforced 
implementation of the curriculum’s techniques in the classroom. 4 

Earlier findings from the Newark study are summarized in this report and new Chicago 
findings are presented in the context of those earlier results. The implementation of the FOL 
model is also described, with a focus on the classroom consultation component. The report 
reviews the extent to which the intervention components were implemented and how the 
preschool context in the two different sites affected that implementation. Finally, impact 
findings from both sites and a cost analysis are presented. 

The report concludes that the FOL intervention model can have a positive impact on 
teachers’ classroom management, can increase instructional time, and can improve children’s 
social and emotional competence during the year of the intervention’s implementation. Howev- 
er, this competence does not appear to lead to children’s improved concurrent academic 
achievement. Longer-term effects of the FOL intervention are inconclusive, given the limited 
data that are available. Similarly, there is no evidence based on these limited data that the 
program’s benefits will outweigh its costs. As discussed in Chapter 6, the implementation and 
impact findings suggest that an FOL-type intervention focused on children’s social and emo- 
tional competence is only one part of creating a quality preschool classroom enviromnent. 


Focus on Children’s Social and Emotional Competence 

Developmental researchers have raised concerns that the accountability measures that have 
accompanied the increase in early childhood services have been too focused on academic readi- 
ness and cognitive outcomes. These researchers suggest that a focus on the social and emotional 
competence of children, and the fundamental skills that underlie this competence, may be an 
important aspect of a high-quality preschool experience. Surveys of teachers echo the need for this 
focus: teachers repeatedly voice that their foremost concern is whether children are prepared 
socially and emotionally for the more fonnal learning environment of elementary school. 5 

Preschool children with behavioral challenges are more likely to face social, behavioral, 
and academic difficulties throughout their school careers than their more well-adjusted peers. In 
addition, children who have difficulty negotiating the classroom enviromnent may disrupt their 
classmates’ academic and social progress. When children act out aggressively or become sad 


4 Webster-Stratton, Reid, and Hammond (2001). 

5 U.S. Department of Education, National Center for Education Statistics (2001). 
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and withdrawn, teachers may be diverted from instructional time to manage these behaviors. 
This can increase teachers’ stress and burnout, as well. 

Whether social and emotional competence matters for later academic skills remains an 
open question. In fact, recent well-designed research has argued convincingly that preschool 
academic skills may be much stronger predictors of later academic achievement than social and 
emotional competence . 6 But social and emotional competence encompasses a number of skills, 
and the recent research suggests that one aspect of this competence — the ability to pay attention 
— does have some predictive power for later academic achievement, more so than measures of 
children’s acting-out behavior or positive social interaction with peers. However, despite its more 
rigorous analytic design (including multiple longitudinal data sets and an extensive set of control 
variables), the research described above is still nonexperimental. Therefore, it requires replication 
in an experimental study such as FOL to most directly test the causal link between changes in 
children’s social and emotional competencies and later academic outcomes. 

Beyond the preschool year, children with low levels of social and emotional compe- 
tence represent an especially large percentage of school districts’ high-expenditure pu- 
pils . 7 These children are more likely to repeat a grade early in elementary school ; 8 they are also 
more likely to receive special education services . 9 In contrast, children who remain curious and 
engaged in school and can maintain positive feelings in the face of academic challenges tend to 
have better school perfomiance and standardized test scores than do other children . 10 


Focus on Classroom Management Strategies 

A number of promising preschool programs have focused on improving children’s social and 
emotional competence. Some models target strategies for teachers, building on the theory that 
teachers’ behavior and interactions play an important role in children’s modulation of their 
attention, emotion, and behavior . 11 A number of smaller studies show that efforts to train 
teachers and parents in these strategies can be effective . 12 Importantly, children do better 
academically when they are enrolled in classrooms that are emotionally more positive and well- 
managed, compared with children who are enrolled in classrooms that are chaotic, disorganized, 


6 Duncan et al. (2007). This research adds to the literature by using a large number of longitudinal studies 
and controlling for an extensive set of characteristics of children and families before preschool. 

7 Chambers, Kidron, and Spain (2004). 

s Beebe-Frankenberger, Bocian, MacMillan, and Gresham (2004). 

9 National Center for Health Statistics (2005); Wagner and Blackorby (2002). 

10 Lepper, Corpus, and Iyengar (2005). 

1 1 Hoglund and Leadbeater (2004); Webster-Stratton, Reid, and Hammond. (2001). 

12 Brotman et al. (2005); Webster-Stratton, Reid, and Hammond (2001). 
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and emotionally negative . 13 In addition, children’s engagement has been found to be higher in 
classrooms where teachers manage behavior problems more effectively . 14 However, these 
findings are also based on correlational data, calling into question whether changing the 
classroom environment will result in changes in children’s behavior and academic skills. The 
FOL study provides an opportunity to test the possible causality of these links. 

In a well-managed classroom, teachers establish clear and firm rules, provide a high 
level of monitoring, and follow a set of simple, behaviorally oriented steps to minimize chil- 
dren’s disruptive behavior . 15 Teachers also need to be flexible in their use of rewards and 
sanctions, recognizing children’s positive behavior with praise, greater responsibility, and 
choice, while responding to disruptive behavior in ways that do not inadvertently reinforce 
children for acting out . 16 In addition to providing well-paced and challenging activities, effective 
teachers maintain a positive climate for learning that is characterized by positive feedback for 
children’s appropriate behavior, established classroom routines with a minimum of disruptive- 
ness, and short transitions between well-planned activities . 17 In poorly managed classrooms, 
teachers may lose large amounts of time to transitional activities such as lining up, putting 
materials away, and moving between small- and large-group activities . 18 

Despite this growing body of evidence, access to high-quality professional development 
that helps preschool teachers strengthen their classroom management skills is rare. In general, 
professional development for early childhood teachers is fragmented, focusing on breadth rather 
than depth of information . 19 Moreover, it generally lacks connection to and integration with 
teacher-identified classroom practices and needs. Exacerbating these concerns is the fact that the 
lead and assistant teachers in a classroom typically do not attend trainings together and do not 
receive similar training content, so there are few opportunities for co-teachers to discuss new 
practices and share information to successfully implement newly learned strategies . 


The Foundations of Learning Model 

The FOL intervention sought to arm teachers with specific management strategies to handle 
daily classroom challenges in order to improve the social and emotional competence of children 


13 Hamre and Pianta (2005). 

14 La Paro, Pianta, and Stuhlman (2004). 

l5 Arnold, McWilliams, and Arnold (1998); Webster-Stratton (1999). 
16 Webster-Stratton (1999). 

17 Bohn, Roehrig, and Pressley (2004). 

18 La Paro and Pianta (2003); Roth, Brooks-Gunn, Linver, and Hofferth (2003). 
19 Bowman, Donovan, and Bums (2001). 
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through the quality of teacher-child interactions. 20 


The intervention delivered four components throughout the preschool year: 

• Teacher training: Lead and assistant teachers were invited to attend five 
Saturday training sessions. The workshops were an adapted version of The 
Incredible Years curriculum developed by Dr. Carolyn Webster-Stratton. 21 
The workshops provided instruction on how to foster positive relationships 
with children; they presented classroom strategies that teachers could use, 
such as setting clear rules; and they provided teachers with techniques to de- 
velop children’s social skills, anger management, and problem-solving abil- 
ity. These strategies represent widely accepted best practices for preschool 
teachers and reflect good classroom management. 

• Classroom-level consultation: To complement the training, teachers were 
assigned a master’s-level Clinical Classroom Consultant to work with them 
in the classroom one day per week throughout the school year. The CCCs 
built collaborative relationships to help model and reinforce the content of 
the training sessions. 

• Stress management: In the winter, teachers participated in a 90-minute stress 
management workshop at their program sites. CCCs also helped support the 
teachers’ use of stress management skills and techniques throughout the year. 

• Individualized child-centered consultation: Beginning in the spring, the 
CCCs provided one-on-one clinical services for a small number of children 
who had not responded sufficiently to the teachers’ improved classroom man- 
agement. By design, the individualized clinical consultation was delivered only 
after children had ample time to react to the new teaching strategies. 

Chapters 2 and 3 discuss in more detail the intervention components and their imple- 
mentation in the Foundations of Learning demonstration. 

Figure 1.1 illustrates the theory of change underlying the FOL model. As depicted, the 
primary focus of all the components of FOL is teachers’ positive classroom management skills. 
Developing these skills is at the core of The Incredible Years training, which is reinforced by 
the CCCs’ modeling and coaching sessions with teachers. By improving teachers’ ability to 


20 Raver et al. (2008), (2009a), (2009b). 

2 'See www.IncredibleYears.com for more information. 
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The Foundations of Learning Demonstration 
Figure 1.1 

Conceptual Model of Foundations of Learning Intervention Effects 


Components of Teacher and Child Outcomes Child Outcomes 

intervention Classroom Outcomes in Preschool in Early Elementary School 



NOTE: Primary outcomes are indicated with a solid outline and secondary outcomes are shown with a dotted 
outline. 


manage the classroom and model positive interactions for students, downtime in classrooms is 
expected to be reduced and instructional time would correspondingly be increased. 

Less directly, it was hypothesized that FOL might change not only the amount but also 
the quality of instructional time, meaning that improvements in classroom climate and chil- 
dren’s behavior might facilitate better teaching. However, there was also the possibility that the 
increased attention by teachers to classroom management might divert their attention from 
providing instruction in academic skills like reading and math. 

The changes in the interactions between teachers and children were expected to im- 
prove a set of skills — collectively referred to as “social and emotional competence” and 
described in Box 1.1 — that enable children to engage positively with peers and teachers and to 
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Box 1.1 


What Is Social and Emotional Competence? 


In this report, the term “competence” is used to indicate a child’s capacity to do something 
and “behavior” is the manifestation of that competency in everyday life. Social and emo- 
tional competence comprises a smaller set of discrete skills, such as the capacity to control 
negative emotions, express emotions, and communicate with peers. Social and emotional 
competence is thought to underlie children’s behaviors, particularly in two areas that are 
thought to be central to longer-term success in school: (1) social behaviors, or children’s 
positive interactions with peers and teachers; and (2) approaches to learning, or children’s 
ability to focus their attention and behavior during classroom activities. 

Strongly related to social and emotional competence is executive function. Its underlying 
skill s are the ability to shift attention, inhibitory control (children’s ability to control their 
immediate or automatic response in favor of a planned response), and working (or short- 
term) memory. Executive function skills are thought to underlie children’s approaches to 
learning as well as their academic achievement. 


focus their attention and behavior during classroom activities. FOL’s particular focus on 
classroom management most directly targets children’s problem behaviors (acting out and 
withdrawal) and the way that they approach learning (their self-regulation and their engagement 
in the learning tasks of preschool). In addition, the CCCs’ one-on-one consultations with 
selected higher-risk children could further reduce problem behavior of children who needed 
more attention beyond the larger classroom management strategies. 

Notably, the intervention’s impact on other components of social and emotional compe- 
tence, such as positive social behavior and appropriate social problem-solving strategies, would 
generally be expected to occur through improvement in more proximal classroom and child 
targets of the intervention (positive classroom management, problem behaviors, and approaches 
to learning). Similarly, gains in the early academic skills of children could occur through their 
improved engagement in classroom activities and the increase in quality instructional time. 
Therefore, these are all considered secondary outcomes, as illustrated in Figure 1.1. 

As discussed in more detail in Chapter 4, another attribute that is strongly related to so- 
cial and emotional competence and its associated behavior is executive function, which includes 
the interrelated skills of attention, inhibitory control (children’s ability to control their automatic 
response), and working (or short-term) memory. Competence in executive functioning is 
thought to be positively associated with children’s approaches to learning as well as learning 
outcomes such as achievement. Both outcomes are reflected in Figure 1.1. 
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Finally, FOL’s benefits were also hypothesized to extend over the long term. Although 
not shown in the model, teachers may carry over their stronger classroom management skills 
beyond the intervention year, leading to a more positive, better managed environment in future 
years. The children could build on their improved social and emotional competence, so that in 
kindergarten and beyond they will have increased ability to regulate their emotions, have more 
positive relationships with their teachers, and perform better on academic outcomes such as 
achievement (grades and test scores), progress (grade-to-grade promotion), and academic parti- 
cipation (attendance). Finally, the children who were more inclined to exhibit problematic 
behavior in preschool may require less disciplinary action and fewer referrals for special services 
as they make the transition to elementary school. However, all of these longer-term effects could 
be influenced by such factors as the quality of the elementary schools that children enter. 


Evolution of the FOL Intervention Model and Research 

The FOL demonstration adopted the model used by the University of Chicago-based CSRP 
(formerly known as the Chicago School Readiness Project). 22 Designed and developed by Dr. 
Cybele Raver, who is a member of the FOL research team, CSRP operated in 18 Head Start 
sites in high-poverty Chicago neighborhoods from 2004 to 2006. (Head Start, the largest 
federally funded early childhood education program in the United States, provides comprehen- 
sive services to low-income children and their families.) The evaluation of CSRP used a 
rigorous research design, in which the 1 8 sites were randomly assigned to one of two groups: 
half received the multicomponent CSRP intervention; the other half, which did not participate in 
the intervention, served as a control group. 23 With random assignment, differences in outcomes 
observed during follow-up can be confidently attributed to the program rather than to preexist- 
ing differences between the two research groups. Such differences, or impacts, that are statisti- 
cally significant are unlikely to have arisen by chance. 

The early results from the CSRP evaluation indicated that the intervention improved the 
quality of the classroom environment as well as preschool outcomes for children. Encouraged 
by these findings, MDRC decided to test the model further. Following a feasibility study and a 
year-long pilot phase, MDRC proceeded with a demonstration of FOL in Newark, New Jersey, 
during the 2007-2008 school year and returned to Chicago for the 2008-2009 school year. (See 
Table 1.1.) FOL, in contrast to CSRP, was intended to be a more real-world test of the interven- 
tion model: the classroom consultation was to be delivered by local social service agencies 

22 CSRP is not associated with The Chicago School®, which is a trademark of The Chicago School of 
Professional Psychology. 

23 CSRP included a teacher’s aide in control classrooms; the presence of the teacher’s aide in control class- 
rooms ensured that any positive impacts of the intervention were not attributable to an improved adult-child 
ratio in the CSRP classrooms. 



The Foundations of Learning Demonstration 
Table 1.1 

Evolution of the Foundations of Learning Demonstration 


Characteristic 

CSRP 1 ' 

FOL Pilot 
in Newark 

FOL Full Scale 
in Newark 

FOL Full Scale 
in Chicago 

School years 

2004-2006 

2006-2007 

2007-2008 

2008-2009 

Total number of sites 

18 

17 

51 

20 

Program sites 

9 

9 

26 

10 

Control sites 

9 

8 

25 

10 

Total number of classrooms 

35 

17 

51 

40 


SOURCES: Raver et al. (2008) and MDRC calculations from random assignment. 
NOTE: a CSRP was formerly known as the Chicago School Readiness Project. 


rather than through a university and, to the extent funding permitted, across a larger sample of 
classrooms and centers. Overall, similar results were anticipated across the CSRP, FOL- 
Newark, and FOL-Chicago studies. 

Two earlier reports concluded that FOL was well implemented in Newark and had 
promising effects during the preschool year in creating a positive classroom environment. 24 The 
program also had positive impacts on teachers’ productive use of classroom time and on certain 
outcomes for children. These findings are discussed further in Chapter 4. 

CSRP, FOL, and other evaluations are part of an emerging body of research on the po- 
tential short- and long-term benefits of addressing the social and emotional competence of 
preschool children through classroom interventions. For example, the nationally representative 
Head Start CARES (Classroom-based Approaches and Resources for Emotion and Social skill 
promotion) Project is currently testing the effects on classrooms, teachers, and children of three 
social-emotional programs implemented in Head Start settings. Funded and conceived by the 
U.S. Department of Health and Human Services, Head Start CARES is being conducted by 
MDRC in collaboration with a group of leading academic experts. 


The Newark and Chicago Contexts of the FOL Intervention 

The fact that the FOL demonstration was implemented in two cities with distinct preschool 
systems provides an opportunity to leam about associations among context, implementation, 
and outcomes. At the time of FOL’ s implementation, Newark was ahead of most of the country 

24 Morris et al. (2010); Lloyd and Bangser (2009). 
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in its implementation of structural changes to promote quality in preschools. The New Jersey 
Supreme Court’s landmark decisions in the Abbott v. Burke class action case required the state 
to increase education funding to disadvantaged districts. In short, the Court ruled that the State 
must provide additional resources, oversight, and regulations for 3 1 low- income school districts 
in New Jersey, including Newark. These requirements created a higher base of resources than is 
available in more typical urban districts. The Abbott mandates included smaller class size 
(limited to 15 students), a lead and assistant teacher in each classroom, higher teacher-student 
ratios (with two teachers per classroom), increased teacher salaries, and stricter teacher creden- 
tialing, among other features. 25 However, even though “Abbott preschool classrooms” received 
substantial resources, initial pilot-testing of FOL by MDRC determined that there remained a 
need for resources to address the social and emotional needs of children. 26 

Chicago FOL preschools represented a more typical urban preschool environment, with 
lower educational requirements for teachers and larger average class sizes. All participating 
centers were part of the Head Start system and were held to national Head Start regulations. At 
the time of the demonstration, this included a minimum educational credential of a Child 
Development Associate degree for lead teachers and a mandated child-teacher ratio of 20:2. 
However, implementation research and reports from the CCCs indicated that staffing shortages 
were common. Given these differences, it was originally hypothesized that Chicago classrooms 
may benefit more from an intervention like FOL. However, findings from an analysis of 
qualitative data from interviews and focus groups with teachers and CCCs in Chicago showed 
that resource and staffing constraints could affect the intervention’s implementation. 


Research Design and Sample 

In Newark, 5 1 preschools were recruited from among the three primary types of venues in the 
city — Head Start centers, community-based child care centers, and public schools. In each 
preschool, one classroom serving primarily four-year-olds was selected to participate in the 
study. Preschools were grouped together by venue, raciaPethnic composition, and neighbor- 
hood, and then randomly assigned within each group. Twenty-six preschools were randomly 
assigned to receive the FOL intervention (referred to throughout this report as the program 
group), and 25 were assigned to the control group, which operated regular preschool classrooms 
in accordance with the Abbott requirements but without the FOL enhancements. 

In Chicago, 20 Head Start centers were recruited to participate in the demonstration, 
and two classrooms in each center were selected for the study. Sites were grouped into pairs 


25 New Jersey Department of Education (2008). 
26 Lloyd and Bangser (2009). 
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based on their matching on key characteristics of teachers (for example, race/ethnicity, teacher 
pay, teacher credentials), families (for example, percent of single-parent families, income level 
of families), and site-level resources (for example, whether the site had an additional staff 
member). Within each pair, one center was randomly assigned to the program group and one to 
the control group. This paired random assignment reduced the likelihood that differences 
between program and control groups would occur by chance. Sites that were assigned to the 
program group received training and coaching in the FOL model; sites that were assigned to the 
control group received Head Start services as usual. 27 

Most important, as was the case in CSRP as well, random assigmnent “worked” in the 
Newark and Chicago FOL studies, in that there were no systematic baseline differences be- 
tween program and control groups in teacher, classroom, and child characteristics. 

Table 1.2 shows a selection of teacher and child characteristics that were collected 
across the FOL-Newark and FOL-Chicago sites. (See, also, Appendix Tables A.l and A.2.) The 
top panel of Table 1.2 presents characteristics of teachers and the bottom panel presents charac- 
teristics of children. The majority of teachers across the two sites were female, African- 
American, and around 40 years of age. In Newark, consistent with the Abbott regulations, all 
lead teachers held a bachelor’s degree or higher, while only about two-thirds of lead teachers in 
Chicago reported a similar level of education. Slightly more than half of the lead teachers in 
Newark reported that they had been teaching in a preschool setting for six years or more, 
compared with nearly 80 percent of the lead teachers in Chicago. The majority of children in 
Chicago (about 70 percent) were African-American, while in Newark the children represented a 
mix of African-American, Hispanic, and Portuguese-speaking (included with “White/Other”). 
More children came from single-parent households in Chicago (92 percent) than in Newark (48 
percent). Overall, the preschools in the sample serve high proportions of low-income families. 

The study of the process and quality of FOL implementation used a mixed-methods ap- 
proach in which measuring the dosage, or amount of the intervention delivered, was comple- 
mented by qualitative data. Throughout this report, implementation refers to the provision of the 
training, consultation, and stress management services and does not include the teachers’ use of 
the strategies learned through the intervention. Dosage data were collected on a regular basis 
throughout the intervention year for the training, classroom-level consultation, stress manage- 
ment, and individualized child-level consultation components of the intervention. Training 
quality was assessed through observation of each training session as well as participant satisfac- 
tion surveys that were completed after the session. The quality of the classroom consultation 
component was assessed through an end-of-the-year survey completed by teachers, weekly 
service provision fomis completed by the CCCs, and weekly technical assistance calls. Stress 

27 For a description of what Head Start services include, see http://eclk.ohs.acf.hhs.gov/hsk/hs/about. 
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The Foundations of Learning Demonstration 

Table 1.2 


Baseline Characteristics of Teachers and Students, Newark and Chicago 



Newark (2007-2008) 

Chicago 

(2008-2009) 

Characteristic 

Program 

Group 

Mean 

Control 

Group 

Mean 

Program 

Group 

Mean 

Control 

Group 

Mean 

Lead teachers 

Female (%) 

88.5 

88.0 

95.0 

100.0 

Age (years) 

37.0 

38.2 

40.2 

47.3 ** 

Race/ethnicity (%) 

Black/ African-American, not Hispanic 

52.0 

66.7 

65.0 

66.7 

Hispanic 

23.1 

17.4 

25.0 

22.2 

White/Othcr 

24.0 

14.3 

0.0 

5.6 

Taught preschool for 6 years or more (%) 

53.8 

56.5 

80.0 

77.8 

Holds bachelor's degree or higher (%) 

96.2 

100.0 

70.0 

61.1 

Students 

Female (%) 

48.6 

48.0 

54.7 

47.5 * 

Age (years) 3 

4.4 

4.3 * 

4.1 

4.1 

Race/ethnicity (%) b 

Black/ African-American, not Hispanic 

42.2 

43.7 

67.7 

72.9 

Hispanic 

35.8 

34.4 

30.1 

26.4 

White/Othcr 

9.9 

10.3 

2.3 

0.8 

Single-parent household (%) 

47.8 

50.0 

92.0 

91.0 

Primary language spoken at home is Spanish (%) 

18.2 

17.5 

29.7 

23.9 

Sample size 

Teachers 

26 

25 

20 

18 

Students 

319 

304 

298 

282 


SOURCES: MDRC calculations from responses to teacher and parent surveys and Head Start data. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Means are adjusted for random assignment block and nesting of students within classrooms. 
a Age at start of school year, September 2007 (in Newark) or September 2008 (in Chicago), calculated 
from date of birth. 

b Race/ethnicity is not available for all students. 
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management quality was assessed through an end-of-workshop, close-ended survey. Finally, 
classroom observations as well as a series of focus groups and one-on-one interviews with 
teachers, CCCs, and district-level educational staff provided additional data on the individual, 
classroom, and site-level experiences that might affect implementation. These findings are 
discussed in Chapter 3. 

While originally designed to collect the same extensive quantitative data on classrooms, 
teachers, and children across both FOL sites, the measurements used in the two FOL studies 
differed somewhat because of funding constraints. The research team chose measurements that 
were complementary and built on the findings from CSRP. FOL-Newark focused on under- 
standing the extent to which FOL changed the classroom environment, by expanding the 
classroom dimensions that were measured in CSRP and assessing child behavior through 
observations in the classroom context. As FOL moved to Chicago, the focus turned to direct 
assessments of children’s social, behavioral, and cognitive skills in order to understand the 
effects of the program model on attributes that children can carry beyond the classroom. These 
direct assessments are currently considered the most reliable measures of child outcomes in 
preschool studies. The measures are discussed in greater detail in Chapter 4. 


Organization of the Report 

The remainder of the report is organized as follows: 

• Chapter 2 describes the components of the FOL intervention, including an 
in-depth look at the classroom consultation component. 

• Chapter 3 provides details on the context, dosage, and quality of FOL’s im- 
plementation in Newark and Chicago. 

• Chapter 4 presents the impacts of FOL on classroom- and child-level out- 
comes for both Newark and Chicago. 

• Chapter 5 discusses the quantifiable costs and benefits of FOL. 

• Chapter 6 summarizes the findings and lessons from the FOL demonstration. 
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Chapter 2 

Design of the Foundations of Learning Intervention 


This chapter presents the rationale for the Foundations of Learning (FOL) program components, 
explaining how they fit together in an integrated model to promote a high-quality preschool 
experience. The multifaceted FOL intervention relied on Clinical Classroom Consultants 
(CCCs) to provide intensive support to teachers and children in order to improve the classroom 
environment and increase children’s social and emotional competence. The strong current 
interest in classroom coaching and consultation services as a way to strengthen teacher practices 
in early childhood education settings includes questions about how best to implement coaching 
models and the most appropriate dosage (or amount) of coaching services. 1 In response, this 
chapter presents an in-depth review of the FOL classroom consultation component, including 
the design, implementation, and monitoring of the consultation process. 


The FOL Components 

FOL focused primarily on enhancing the quality of teacher-student interactions as a means of 
promoting young children’s social and emotional competence. The intervention was designed and 
sequenced, as shown in Figure 2.1, to support teachers’ use of positive classroom management 
skills. The core of the intervention was an adapted version of The Incredible Years curriculum 
developed by Dr. Carolyn Webster-Stratton. 2 The Incredible Years is a classroom management 
curriculum specifying the implementation of strategies that are based on early childhood best 
practices. While the strategies are simple and generally familiar to teachers, they are designed to 
be implemented in a specific sequenced and staged manner. The teacher training and other 
components of the FOL intervention are described briefly below, with an extended discussion of 
the design of the classroom consultation component in the second half of the chapter. 

Teacher Training 

Since teachers typically have limited practical training in how to promote children’s so- 
cial and emotional competence, 3 the FOL intervention included workshops for teachers on how 


'For information about classroom coaching and consultation in early childhood education settings, see 
Lloyd and Modlin (2012). For implementation of coaching models, see Smith et al. (2012). For dosage of 
coaching services, see Wasik, Mattera, Lloyd, and Boiler (in press). 

2 See www.IncredibleYears.com for more information. 

3 Weist (2005); Raver, (2002). 
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The Foundations of Learning Demonstration 
Figure 2.1 

Intervention Timeline 


Sept. 

Oct. 

Nov. 

Dec. 

Jan. 

Feb. 

Mar. 

Apr. 

May 

June 


Monthly training sessions for teachers 


CCCs enter 
classrooms 


Classroom consultation by CCCs 


Stress management 
workshop 


One-on-one services for 
children 


NOTE: CCC = Clinical Classroom Consultant. 

to establish positive relationships with their students, encourage age-appropriate behavior, and 
minimize classroom disruptions and confrontation. 4 Lead and assistant teachers were invited to 
attend five Saturday training workshops, once a month for six hours each, from late September 
to January. The workshops instructed teachers in how to develop positive relationships with 
children and their families; present classroom strategies that teachers could use, such as setting 
clear rules, outlining predictable limits, and instituting a discipline structure that minimizes 
classroom disruptions and avoids confrontation; and equipped teachers with techniques to 
develop children’s social skills, anger management, and problem-solving ability. Box 2.1 
outlines the content that was covered during the training sessions. The model stipulates that the 
strategies be used within a sequenced hierarchy starting with building positive relationships 
(which are the foundation for proactive classroom management) and moving to more child- 
specific and targeted practices such as praise, encouragement, limit-setting, and consequences 
after the foundation has been set in place. Figure 2.2 presents an adapted version of The Incred- 


4 The workshops are an adapted version of The Incredible Years teacher training series curriculum devel- 
oped by Dr. Carolyn Webster-Stratton. See www.lncredibleYears.com for more information. 
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Box 2.1 


Overview of Teacher and Clinical Classroom Consultant Training Content 

Training 1: Building Positive Relationships with Students, Proactive Teaching, and 
Preventative Approaches 

• Promoting positive relationships with students by listening 

• Understanding students’ strengths 

• Collaborating with caretakers to benefit children 

• Creating predictable and safe learning environments 

• Giving clear and specific directions in positive ter ms 

• Helping students make choices 

• Using nonverbal cues and signals as a classroom management strategy 

• Understanding the process, implementation, and benefits of a good behavior plan 

Training 2: Teacher Attention, Praise and Encouragement, Teacher Coaching, and 
Child-Directed Play and Friendship Skills 

• Enhancing children’s self-esteem and confidence by using praise 

• Praising positive social and academic behaviors 

• Using descriptive comments to facilitate children’s social learning 

• Developing children’s abilities to praise themselves and their peers 

• Helping children appreciate the achievement of others 

Training 3: Motivating Students Through Incentives and Decreasing Inappropriate Behavior 

• Understanding and effectively using the classroom discipline plan, or hierarchy of consequences 

• Understanding the advantages and disadvantages of using incentives 

• Setting up and using developmental^ appropriate incentives 

• Using incentives to facilitate children’s internal learning processes 

• Using logical consequences to foster appropriate behavior 

Training 4: Teaching Children to Be Socially Competent 

• Increasing children’s awareness of different feelings 

• Building children’s emotional vocabulary 

• Understanding children’s anger and helping them to manage it successfully 

• Promoting children’s responsibility in the classroom 

• Fostering positive reputations 

Training 5: Training Review and Wrap-Up 

• Overview of previous training content 

• Targeted review of previous topics or training content that teachers request 
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■Order of Implementation' 


The Foundations of Learning Demonstration 


Figure 2.2 

The Incredible Years Teaching Pyramid 



SOURCE: Adapted from The Incredible Years Teaching Pyramid. See www.IncredibleYears.com/Resources/ 
teacher-program-pyramid.pdf. 
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ible Years Teaching Pyramid, 5 which illustrates the sequence of classroom management 
strategies. 


Classroom Consultation 

Training alone is often insufficient to ensure that teachers implement new practices. 6 
Therefore, the FOL program design called for a master’s-level CCC to provide in-class support 
one day per week to reinforce the lessons from the training, to model appropriate techniques, 
and to act as a sounding board for teachers. This regular in-class presence, which began even 
before the teacher training workshops started, contrasted with the “on call” approach used in 
many programs. 7 In the FOL study, master’s-level CCC Coordinators were responsible for 
overseeing training and support efforts for the CCCs. 

Stress Management 

In addition, because stress in teachers’ personal and professional lives can reduce their 
effectiveness, 8 teachers attended a stress management workshop in January or February that was 
customized to respond to the lead and assistant teachers’ particular needs. The 90-minute 
workshop was delivered conveniently on-site in the teachers’ school or center. The CCC also 
provided support to the teachers throughout the year on issues related to stress and burnout. 

Individualized, Child-Centered Consultation 

Finally, the FOL intervention recognized that teachers’ improved classroom manage- 
ment skills (the universal component of the model) might not be enough to support the social 
and emotional needs of all children in the classroom. 9 Therefore, the CCC provided individual- 
ized consultation services to children who needed additional support. However, the clinical 
services were deferred until the final third of the school year (March through May) to ensure 
that the universal component was implemented fully and the children had sufficient time to 
respond to the teachers’ new strategies. In addition to providing direct support to children, the 
insights that were gathered from the individualized consultation sessions were shared with 


5 See www.IncredibleYears.com/Resources/teacher-program-pyramid.pdf. 

6 Raver (2002); Raver et al. (2008); Webster-Stratton, Reid, and Hammond (2001); Neuman and Wright 
( 2010 ). 

7 For a detailed discussion of the programs reviewed, see Alkon, Ramler, and MacLennan (2003); Farmer- 
Dougan, Viechtbauer, and French (1999); Gilliam (2007); Hennigan, Upshur, and Wenz-Gross (2004); Kagan 
(2003); Lehman et al. (2005). 

8 Curbow et al. (2000); Deery-Schmitt and Todd (1995); Manlove (1993). 

9 There is some evidence of positive results from a “blended” program of teacher training supplemented by 
individualized services for high-risk children in kindergarten and first grade (August, Realmuto, Hektner, and 
Bloomquist, 2001; August, Egan, Realmuto, and Hektner, 2003). 
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teachers in an effort to help them relfame potentially negative perceptions about children who 
displayed particularly challenging behaviors in the classroom. 


Technical Assistance and Oversight 

In Newark and Chicago, the FOL intervention was accompanied by a comprehensive technical 
assistance plan to support the CCCs, teachers, and administrators at program sites. Technical 
assistance efforts were carried out by a core group of New York City-based project staff at 
MDRC. The primary technical assistance activities included meetings with key early childhood 
education stakeholders, observations of FOL training activities, periodic site visits to FOL 
program classrooms, and supervision of consultation staff through regular meetings and review 
of consultation documentation. 

Meetings with Key Stakeholders 

The early childhood education systems in both Newark and Chicago included multiple 
agencies that were responsible for funding, monitoring, and providing support services to the 
schools and community-based organizations that operate early childhood education programs. 
(See Appendix Table B.l.) For example, the early childhood education systems in Newark and 
Chicago included on-site special education services and mental health consultation for children, 
as well as professional development (for example, mentors for teachers). These support services 
were provided by field-based professionals who were employed by the local school district, a 
local government agency, or a private contractor. Meetings were held with key stakeholder 
groups to inform them of the plans for the FOL demonstration and to ensure that professionals 
in the field, such as special education providers, understood the purpose of the intervention to 
ensure that teachers would not receive infonnation that conflicted with the FOL approach. 

Observations of Training 

Members of the technical assistance team at MDRC attended all of the intervention 
training workshops to track teacher attendance and to observe teachers’ responses to the 
information that was presented. Information that was gathered during the observations was 
shared with CCCs and trainers during debriefing meetings after each training session. 
Debriefing meetings included discussions about the session content and quality and aspects 
of the session that worked particularly well or poorly. CCCs were then tasked to work with 
any teachers who missed the training session to bring them up to speed on the training 
content. In addition, CCCs attempted to understand and help resolve teachers’ barriers to 
attending the workshops. 
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Site Visits 


Site visits to program classrooms provided the technical assistance team with the 
opportunity to observe FOL implementation first-hand and solicit feedback from teachers 
about their experiences with the intervention, including their work with the CCCs, their 
reactions to the training content and its delivery, their implementation of The Incredible 
Years strategies and techniques, any issues that constrained intervention implementation, 
and data collection procedures. Site visits also provided an occasion for the technical 
assistance team to connect with center directors and school principals to discuss the imple- 
mentation of FOL and understand their operating environments. In general, there were 
monthly technical assistance visits to FOL program classrooms in Newark (with some 
variability by individual site), but there were only two site visits in total to FOL-Chicago 
classrooms because of resource constraints that limited travel. 

Supervision of Consultation Staff 

Supervision of consultation staff was provided in two different fonnats: one-on-one 
clinical meetings and full-team technical assistance meetings. The purpose of the full-team 
technical assistance supervision was to monitor and understand the quality of the consultation 
services that were provided to teachers as well as the teachers’ implementation of the FOL 
intervention (based on CCC reports). The full team meetings were co-facilitated by the CCC 
Coordinator and the MDRC Project Manager. Both levels of supervision are described in more 
detail in the section on classroom consultation that follows. 

The remainder of this chapter describes in detail the classroom consultation model, 
which sets FOL apart from other early childhood interventions or professional development 
programs because of the intensity, duration, and multilevel focus of the classroom consultation 
support. The prescribed approach to classroom consultation as well as the processes for hiring 
and supervising consultation staff are outlined here to provide administrators, school district 
officials, and others with a framework to understand what it takes to bring the FOL consultation 
model to scale in preschool sites. 


An In-Depth Look at the FOL Classroom Consultation 

Interest in classroom consultation as part of professional development efforts for early child- 
hood educators has increased in recent years, both as part of a comprehensive intervention, such 
as FOL, and as a stand-alone effort to increase the quality of instruction and the preschool 
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environment. However, there is currently very limited infonnation about the design of consulta- 
tion models or how effective they are in particular contexts. 10 

The consultation model that was adopted for FOL accounted for the intervention’s goals 
and the needs of the participating preschool classrooms, including the selection and training of 
CCCs, expectations for the consultation process, and supervisory oversight and support. 

FOL consultation differed in many respects from typical early childhood coaching 
models: 11 (1) it was multilayered, providing support to both adults and children; (2) it was 
preventive, with both a programmatic and clinical focus designed to change teachers’ practice 
and, later in the year, to change the behavior of selected children; (3) it was time-intensive, 
spanning the entire academic year and including a full day per week of in-classroom consulta- 
tion; (4) it aimed to be culturally sensitive by including deliberate raciaFethnic matching of 
CCCs to the teachers and children they serve; and (5) CCCs were hired outside of the preschool 
systems in which they were working and were highly skilled clinicians with master’s degrees. 12 

Consultant Qualifications 

The CCC’s role was complementary, but distinct, and required hiring individuals with a 
wide range of knowledge, skills, and experience. In FOL, individuals with expertise in early 
childhood were sought, but the project also prioritized recruitment and hiring of individuals 
with strong interpersonal and clinical skills because of the importance of (1) building positive 
working relationships between CCCs and teachers; (2) promoting productive working relation- 
ships among teaching partners in the classrooms; and (3) working one-on-one in a clinical 
capacity with individual children. In addition, the project sought individuals who were cultural- 
ly sensitive, had experience supporting adult learning, and had the ability to work in complex 
early childhood settings in urban environments. 

Recent graduates with a master’s degree in social work, psychology, and/or counseling 
were considered to be ideal because they were thought to have the clinical skills needed to 
support teachers and children. They were also thought to be more open to the process of 
learning and taking on a new intervention, since they were likely to be less focused than veteran 
mental health professionals might be on one particular method of practice or service delivery. 
Box 2.2 presents the job description and qualifications that were used to recruit CCCs. Appendix 


10 For information about the design of consultation models, see Lloyd and Modlin (2012); and Sheridan, 
Edwards, Marvin, and Knoche (2009). For information about various models’ effectiveness in particular 
contexts, see Zaslow et al. (2010). 

n For a review of coaching models, see Alkon, Ramler, and MacLennan (2003); and Duran et al. (2009). 
12 CCCs were employed by local, independent agencies for each demonstration that were not a part of the 
school district. 
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Box 2.2 


Clinical Classroom Consultant Job Description 

The job description and qualifications presented below were used to recruit clinical class- 
room consultants (referred to here as “mental health consultants”) for the FOL-Newark 
and FOL-Chicago demonstrations. Bracketed text has been added for clarification. 

Mental health consultants will provide and assist in the implementation of research-based 
behavioral strategies and classroom interventions that are geared toward reducing children’s 
problem behavior in the classroom and supporting teacher’s roles in this process. To 
achieve this goal, successful candidates will both attend and then apply behavioral modifi- 
cation/training to urban early childhood classrooms using a consultation model. This on-site 
presence supports teachers in handling children’s challenging and disruptive behaviors, and 
provides teachers with concrete problem-solving skills to ensure that all children in the 
classroom are better able to learn. 

Essential Functions 

• 40 percent: Provide services to teaching staff in early educational settings, following a 
manualized consultation approach [that is, the protocol for consultation services] that 
includes helping teachers to implement strengths-based improvements in instructional 
practices [that is, drawing on teachers’ individual strengths], providing behavioral and 
stress management training for teachers, attending weekly clinical supervision, and re- 
search based staff meetings. 

• 40 percent: Participate in data collection efforts by completing weekly documentation 
for both teachers and provide classroom-based interventions for children unresponsive 
to teachers’ efforts at implementing behavioral management strategies. Participate in 
documenting the process of implementing the intervention. 

• 20 percent: Sustain the mental health needs of children through capacity building, 
including creating treatment plans co-jointly with parents, teachers, and other profes- 
sionals to monitor children seen for direct intervention, and referring children to com- 
munity-based mental health agencies. 

Qualifications 

• Master’s degree from an accredited school of social work, counseling, or psychology 

• Current social work or counseling licensure or board-eligible 

• Commitment to working with ethnically diverse children, families, and professionals 

• Understanding of early childhood development 

• Experience working with families from high-risk, low-income, and urban backgrounds 

• Ability to work independently and within a team 

• Comfort in working in a host environment 

• Strong reading, writing, and communication skills 

• Bicultural and bilingual in English and Spanish is highly preferred 

• Valid driver’s license 

• Access to own transportation necessary 
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Table B.2 presents the CCCs’ demographics and characteristics for the FOL-Newark and FOL- 
Chicago demonstrations. It was challenging to recruit CCCs with the broad set of skills desired 
for FOL implementation. Recruitment was made all the more difficult because the project was a 
time-limited research initiative. 

Training 

Once hired, CCCs were provided with an initial orientation to their roles and the pro- 
ject, in addition to training on The Incredible Years curriculum. This orientation occurred 
approximately one week before the start of the school year and was designed to provide CCCs 
with a basic understanding of the intervention and their role in it. The orientation was also 
structured to facilitate a sense of camaraderie and rapport among the CCCs, including ice- 
breakers, interactive exercises, and multiple opportunities for CCCs to share, reflect, and ask 
questions. 

The orientation of CCCs was a two-day seminar for FOL-Newark but, based on the 
Newark experience, was expanded to a three-day seminar for FOL-Chicago. The Chicago 
training included all of the Newark content as well as additional topics, such as the CCCs’ role 
with respect to the intervention’s implementation and the research initiative. 

Box 2.3 provides an overview of the various components of the CCCs’ training. 

In advance of the orientation, CCCs were given a description of the research study, the 
book How to Promote Children ’s Social and Emotional Competence , 13 the intervention manual, 
and the policies and procedures manual for their hiring agency. Once the school year began, 
CCCs joined the teachers at their training workshops on The Incredible Years approach. Trainings 
were held roughly once a month from late September through January of the school year. Co- 
training enabled the CCCs and teachers to interact outside of the classroom; it also ensured that 
both CCCs and teachers received the same information about the intervention, providing them 
with a common language and framework for discussing classroom implementation. 

Consultation Process 

CCCs followed a manual that not only specified the amount of time that they should 
spend in classrooms, but also provided clear guidance about the activities in which they should 
engage. The clinical consultation model specified that CCCs spend six to seven hours per week 
(during one specified weekday) on-site at the participating preschool programs and outlined a 


13 Webster-Stratton (1999). 
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Box 2.3 


Overview of Training Content for Clinical Classroom Consultants 

Training Day 1 

Overview of MDRC, the Hiring Agency, and the Study. Covered the purpose and mission of 
MDRC, information about the consultant hiring agency, and a review of the Foundations of 
Learning project. 

Staff Roles and Responsibilities. Overview of the roles of all staff associated with the project, 
including the research and technical assistance team, supervisory staff, classroom consultants, 
treatment and comparison teachers, and, in Chicago, other professionals in the preschool environ- 
ment such as family workers, resource teachers, and mental health consultants. 

FOL Timeline and Model Components. A review of the research project, the program model, and 
the intervention timeline. Particular attention was paid in the Chicago orientation to the relationship 
of the intervention, the consultant role, and the data collection processes. 

City and Preschool Contexts. History of the cities in which the intervention was being imple- 
mented and an overview of the preschool contexts in each city. Tips for working within preschool 
settings were also provided. 

Training Day 2 

Intervention Implementation and Maintaining Fidelity. Clarification of the steps that were 
necessary to ensure that the intervention was implemented as intended and including a review of 
the various phases of the consultant model and the expectations around dosage for teachers and 
children. 

Practical Application of the Model. CCCs used the CCC manual as a framework for reviewing a 
“typical” day in the life of a consultant, including the types of issues CCCs might face and expec- 
tations for their role in the project, such as the phases of the consultant’s role and activities to 
complete while working in classrooms. 

Modeling and Role Play. Using field-based scenarios derived from FOL-Newark and CSRP 
(formerly the Chicago School Readiness Project), this aspect of the training provided an opportunity 
for consultants to role-play challenging situations they might encounter in their work. Time was built 
in to reflect on and refine their practice based on feedback and guidance from the administrative and 
supervision teams as well as their peers. 

Training Day 3 

Intervention Documentation. A line-by-line review of the data collection instruments. Consult- 
ants also practiced completing the forms and used feedback from the orientation facilitators to 
refine their documentation skills. 

Site Assignments. Consultants received their assigned list of sites; spent time as a group touring 
the neighborhoods, centers, and schools where they were assigned; and reflected on their observa- 
tions and any concerns with the team. 

Evaluation. CCCs provided written feedback on the orientation. 
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progression of phases for the CCC’s role, as explained in Box 2.4. The Incredible Years 
curriculum developers supplement teacher training with a mentoring process as part of the 
intervention model; however, consultation in FOL was designed to be more intensive than the 
model that typically accompanies the curriculum. The intended dosage and duration of class- 
room consultation were quite high by design because CCCs were expected to take on multiple 
roles and perform several tasks during each consultation session. This also served to establish 
consistency in the provision of consultation services, and gave CCCs the time to develop a clear 
understanding of the intricacies of each classroom’s environment. CCCs were expected to take 
into account variations in program context and teachers’ responsiveness, with deviations from 
prescribed guidelines made in consultation with their supervisors and the MDRC technical 
assistance team. 

CCCs entered the classrooms in September. As part of the entry and joining phases, 
they were expected to spend about a month becoming familiar with the site and surrounding 
environment as well as establishing positive relationships with the teachers, administrators, and 
other key early childhood players such as mental health consultants. During this time, the CCCs 
did not engage in active consultation; their primary task was to acclimate themselves to class- 
room routines and norms, while also allowing teachers to become more familiar with them and 
the project. CCCs were advised to support teachers with basic classroom tasks by assisting with 
meals, clean-up, and set-up of daily materials. This support was expected to continue through- 
out the year and was designed to help ease the transition of the CCCs into the classrooms, 
increasing the potential for the development of trusting working relationships between CCCs 
and teachers. 

After the first teacher training session in late September, CCCs were expected to sup- 
port teachers in their efforts to incorporate the strategies from the training into the classroom 
environment. Promoting implementation of the strategies involved activities such as reviewing 
infomiation from the training workshops, modeling The Incredible Years techniques, and 
supporting teachers with stress management. Although the consultation support lasted through- 
out the year, the FOL model stipulated that active consultation activities related to implementa- 
tion of The Incredible Years techniques and stress management would be most intensive from 
October to January. CCCs were also supposed to continue providing support on basic classroom 
tasks in an effort to maintain positive relationships with the teachers and children. 

Focused weekly debriefings, conducted over lunch, during students’ rest time, or after 
school, were expected to be a formal part of the consultation process. The debriefings ensured 
that teachers and CCCs were “on the same page” and communicated with each other regularly. 
The debriefings were also a valuable opportunity to review classroom challenges and successes, 
whether the goals of the teachers and the intervention were being met, and ways to adapt the 
classroom management strategies if needed. The debriefings encouraged the teachers to reflect 
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Box 2.4 


Phases of the Clinical Classroom Consultant’s Role 

Pre-entry: Before entering the classroom, the Clinical Classroom Consultant (CCC) needs to 
leam about the preschool site and the community it serves. The information collection process 
should include a visit to the community. 

Entry: Lasting as much as a month, the entry phase begins when the CCC first meets the 
teacher, typically at the preschool site. The key goal for this phase is to lay the foundation for 
trusting relationships with the teachers, other staff, children, and parents by “hanging out, 
helping out, and being checked out.” 

Joining: The goal of joining is not only to enter, but also to become part of the setting. CCCs 
leam from the setting and the people in it, communicate openness to give and take, and share 
information about themselves. 

Assessment: Overlapping with the entry and joining phases, the purpose of assessment 
includes developing a more complete understanding of the strengths, challenges, and re- 
sources of the preschool site, the surrounding community, and the people in them. 

Consulting: This is a structured, collaborative process in which the CCC models and rein- 
forces the lessons from the teacher training to promote the teachers’ effective classroom 
management and children’s social and emotional development. 

Stress and burnout: Throughout the school year, the CCC who is assigned to the classroom 
helps teachers address issues of personal and professional stress. In January or February, the 
CCC Coordinator, often in conjunction with the CCC who is assigned to the classroom, 
conducts an on-site stress management workshop of 90 minutes to 2 hours that is customized 
for the lead and assistant teachers. 

Direct services to children: In February, the teachers and the CCC identify selected children 
for whom the universal classroom management strategies have not been effective. The CCC 
provides individualized attention to help address these children’s continuing social and 
emotional needs. 

Referral: In May, when the FOL program winds down, some of the children who had been 
receiving individualized services are referred for other services in the community. 

Termination: As the intervention year comes to an end, the CCC prepares teachers, other 
staff, children, and parents for the CCC’s eventual departure. This includes addressing any 
unresolved issues and making plans with teachers for ways that they can continue to use their 
newly acquired skills in the future. 

NOTE: This box is adapted from Madison-Boyd et al. (2006). 
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on their own practice, as well as the children’s responses to the changes teachers made in 
their practice. 

The documentation requirements for CCCs supported the supervision and quality as- 
surance processes that are described in the section that follows. CCCs submitted a weekly 
Service Provision Form (SPF) for each teacher . 14 The SPFs were used to direct discussions 
during CCC supervision and served as a data source for monitoring implementation of the 
intervention. CCCs were also responsible for documenting their individual-level clinical 
sessions with children once those sessions were initiated in March. 

CCC Supervision 

An important aspect of the model was ongoing supervision. This was conceptualized as 
a way to support to CCCs individually and as a team, and was also designed to provide pro- 
grammatic updates and to prevent inappropriate deviation from the program model. To accom- 
plish these goals, CCCs received two fomis of supervision: one-on-one clinical meetings and 
full-team technical assistance meetings. 

Clinical Meetings 

CCCs received individualized weekly supervision from a licensed clinical social work- 
er, as well as regularly scheduled group supervision. This supervision was designed to strength- 
en the CCCs’ clinical skills in an effort to help them better serve teachers and children. Topics 
included building relationships with teachers, setting boundaries between teachers and the 
CCCs, time management, navigation of intra- and inter-organizational relationships and politics, 
and therapeutic techniques for one-on-one work with individual children. 

Technical Assistance Meetings 

Technical assistance meetings were conducted by the master’s level CCC Coordina- 
tors and the MDRC Project Manager. The meetings took place in a group format — mean- 
ing that all CCCs and, whenever possible, the CCC Coordinators and Project Manager were 
present. The meeting was structured so that CCCs could report on the work they were doing 
and the teachers’ responses to the intervention. The meeting was held weekly, but because 
of the number of classrooms that were participating in the demonstration, each classroom 
was discussed every other week. 


14 The SPFs were a structured template that provided a platform for CCCs to document and guide their 
work with teachers. See Appendix B for a sample SPF. 
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The focus was on ensuring that the model was being implemented as intended and de- 
ciding whether any deviations from the model were acceptable adaptations to the local imple- 
mentation context. For example, a CCC in a site that experienced significant teacher turnover 
had to return frequently to the “joining” phase of the model to develop relationships with new 
teachers, a prerequisite to active consultation. Immediate problems were identified and handled 
quickly. Notes were taken during each meeting to provide an ongoing record of developments 
in the classrooms. 

Finally, CCCs were supported in completing the programmatic documentation, which 
helped to facilitate fidelity to the model and keep the work on track. The content of the SPFs 
was to be presented verbally in weekly meetings with the Project Manager. The project’s 
administrative assistant was responsible for documenting receipt of the fonns to ensure that (1) 
they were being completed in an accurate and timely manner; and (2) teachers and CCCs were 
meeting as planned. If documentation was missing, the CCCs and the CCC Coordinator were 
contacted to facilitate receipt of the data. 

The following case study presents a hypothetical scenario that illustrates selected phases 
of the consultation process and the CCCs’ roles and responsibilities in the FOL intervention 
classrooms. 


A Day in the Life of an FOL Clinical Classroom Consultant 

Nancy, a CCC, has been in the classroom with Marcia, the lead teacher, and Kim, the assistant 
teacher, for nearly two months. During these initial months, Nancy worked hard to develop solid 
relationships with both teachers. Of critical importance was spending time to ensure a shared 
understanding with them about Nancy’s role, the intervention, and how tilings would unfold over 
the course of the year. Marcia has a bachelor’s degree in early childhood education and more 
than 10 years of classroom experience. She has expressed limited interest in implementing the 
intervention, but is committed to attending the training sessions. She has said to Nancy, “I like 
you, but these strategies can’t tell me anything new. I have been teaching for years.” Kim, the 
assistant teacher, has also been in the preschool field for more than 10 years and in that time has 
earned a Child Development Associate credential. She is excited about the intervention, but as 
the assistant teacher, she lacks the authority and ability to make changes in the classroom. 

On this particular visit, the teachers have set a goal to use circle time (or “group time”) 
to implement “proximity praise” — praising children who are exhibiting the behavior that the 
teachers want to see. While helping the teachers clean up the breakfast tables, Nancy engages 
both of them in reviewing what they plan to do during circle time and how. They also decide 
that Nancy’s role will be only to observe. Nancy plans to sit in the circle with the teachers and 
children to hear and see what is happening, but she will not intervene in the activity in any way. 
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Circle time starts and Kim focuses her attention on Ian, a child who often acts out dur- 
ing the school day. When Kim notices that he is following directions, she initiates the proximity 
praise strategy by commenting, “Great job, Ian. I really like the way you are sitting ‘criss-cross 
applesauce’ with your hands in your own space and your eyes on me.” Marcia follows Kim’s 
comment with one of her own: “I do not like the way Bella is touching others and not paying 
attention to our story.” Bella responds by pouting and turning away from the circle. Nancy, 
meanwhile, is making mental notes of what the teachers say and how the children are respond- 
ing to the teachers’ comments. 

Nancy wanted to hold a debriefing meeting with the teachers right after circle time end- 
ed, but there was not enough time. Kim was asked to cover another classroom where the lead 
teacher was absent, leaving Marcia and Nancy in the classroom until the substitute arrived. This 
was the third time Kim had been pulled out of the classroom during a consultation session. 
Nancy knew she would need to discuss this issue with the center director, since she was not 
supposed to serve as a teacher but now was expected to play that role in Kim’s absence. 

After lunch, Kim returned and Nancy led a debriefing meeting during the children’s rest 
time. Nancy began by reviewing the goal for the use of proximity praise that they had all agreed 
to in the previous week. She then asked both teachers to reflect on how the activity had gone, 
including what went well and what could have gone better. Nancy focused on listening to the 
teachers’ reflections and withheld feedback until each teacher had a chance to share her per- 
spective. Both teachers deemed the technique a success, stating that they implemented proximi- 
ty praise appropriately and with ease, and that the children responded to the technique as they 
expected. Nancy gently challenged them about their perspectives. She shared her observation of 
children who were having trouble paying attention during the beginning of circle time, but who, 
after hearing Kim’s positive feedback directed at Ian, were more engaged in the activity. She 
also shared her observations of Bella’s behavior, wondering if they had noticed the same thing 
and what could be done differently. 

In addition, Nancy gave the teachers an assignment to review a segment from one of the 
training videos that showed proximity praise in action. She also made herself a note to inform 
the trainer about her circle time observations, so that the technique could be highlighted during 
the upcoming training, and other teachers would have the opportunity to share their experiences. 
Finally, a plan was developed for the teachers to continue to try to use proximity praise both in 
and outside of circle time during the rest of the week and to reflect on their experiences with 
each other in Nancy’s absence. 

As the day progressed, Nancy modeled other strategies from the training for the teach- 
ers during small group activities and free play, actively describing how and what she was doing 
as she did it. Toward the end of the day, as the teachers were preparing for the children to be 
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picked up, Nancy encouraged the teachers to complete “Happy Grams,” an Incredible Years 
tool for sharing information with caretakers. She explained that providing “Happy Grams” gives 
families an opportunity to receive positive communication and feedback about children’s time 
at school. For Ian, this was particularly important, since most interactions with his parents had 
been about things that went wrong at school. Engaging in this activity would help lay a founda- 
tion for positive experiences in his educational process. 

Before leaving for the day, Nancy and the teachers quickly reviewed their goals again 
and the teachers agreed to contact Nancy by phone or e-mail if any questions or concerns came 
up before her next visit. Nancy completed her Service Provision Form after leaving the class- 
room, chronicling the day’s occurrences and noting the goals for the following week. 

At the end of the week, Nancy and her clinical supervisor used her SPF to review her 
classroom experiences with Marcia and Kim. Nancy said that she was particularly concerned 
about working with Marcia on managing children’s behaviors, and while she wanted Marcia to 
be more reflective about her use of the strategies, she was concerned about damaging the solid 
relationship they had developed early on in the school year. Nancy’s supervisor suggested that 
Nancy use praise in her work with Marcia, supplementing it with examples of Marcia using The 
Incredible Years techniques successfully. Nancy left supervision with a plan to observe Mar- 
cia’s interactions with the children and to document particularly positive examples that she 
could then use as a way to describe the connection between Marcia’s correct use of The Incred- 
ible Years strategies and the children’s responses to them. She hoped this approach would 
encourage Marcia to be more enthusiastic about implementing the intervention. Nancy would 
report back on how well this strategy went in the next supervision session. 

This hypothetical case study highlights selected phases and activities that are part of the 
consultation process. The CCC spent time joining with teachers, working toward becoming an 
integral part of the classroom. She assessed the environment, including the teachers’ roles, 
weaknesses, and strengths; engaged in active consultation by modeling and reinforcing training 
content; and worked closely with teachers, debriefing with them about their work and setting 
goals. Finally, she spent time in supervision reviewing her work and making plans for subse- 
quent visits. These tasks, while straightforward, were implemented in a fairly complex envi- 
ronment, illustrating the need to understand not only what should be happening, but how 
context can ultimately have an impact on the delivery of consultation services. 


Using data from the FOL-Newark and FOL-Chicago implementation studies, Chapter 3 shows 
how each component of the intervention was delivered in practice. The discussion is framed by 
the context in which implementation occurred, including the level of organizational resources 
and staffing, the delivery of consultation services, and teachers’ receptivity to the intervention. 
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Chapter 3 


Implementation of the 
Foundations of Learning Intervention 


Chapter 2 described the core components of the Foundations of Learning (FOL) intervention, 
which, if implemented well, were expected to lead to direct changes in teachers’ classroom 
practices and, consequently, to positive outcomes for children. The implementation analysis in 
this chapter addresses the dosage (that is, the amount) and quality of the core components that 
were delivered in practice, as well as teachers’ engagement in the implementation process. Pre- 
senting these data in tandem helps to clarify whether the teachers actually understood and inter- 
nalized the classroom management strategies and the factors that facilitated and impeded the 
delivery of the intervention components. 

The analysis also provides an explicit basis for understanding what is needed to support 
implementation of the FOL intervention in distinct settings. Of particular interest are differences 
in the way classroom consultation took place in the two cities, an important facet of the inter- 
vention to understand given the increased interest in and funding of coaching and consultation 
in early childhood settings. 1 


Summary of Findings 

• The core FOL components were delivered at relatively high levels of 
dosage in both Newark and Chicago, two very different contexts. Quali- 
ty was also high as rated by teachers. FOL-Newark was implemented in a 
preschool system that is rich in resources, which is fairly uncommon in large, 
urban educational systems. FOL-Chicago, by contrast, was implemented in 
classrooms with considerably fewer resources as well as staffing shortages. 
Despite the differences in resources, the intervention components were gen- 
erally delivered at high levels of dosage in both settings. Moreover, teachers 
in Newark and Chicago reported similarly high levels of satisfaction across 
the components of the intervention. 

• Clinical Classroom Consultants (CCCs) in Newark consulted with 
teachers on classroom management strategies twice as frequently as 
CCCs in Chicago. Although the overall amount of teacher-focused class- 


'Lloyd and Modlin (2012). 
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room consultation was high in both Newark and Chicago, the focus of con- 
sultation differed somewhat in the two sites. CCCs engaged teachers in 
Newark in implementing classroom management strategies from The Incred- 
ible Years consistently across classrooms. In Chicago, however, the focus on 
classroom management strategies was less consistent across classrooms. 
Analysis of qualitative data from interviews indicates that several factors 
may have contributed to inconsistencies in consultation services that were 
provided to teachers: classroom environment and teacher stress, teacher re- 
ceptivity to new methods, and CCCs’ comfort level in working with teachers. 


Implementation Study Design and Data Sources 

Building on evidence from CSRP, the FOL-Newark and FOL-Chicago studies included imple- 
mentation analyses to shed light on the mechanisms that might have facilitated or impeded 
good-quality implementation. The implementation analysis in FOL was designed to help re- 
searchers understand the delivery of the individual intervention components. The studies used a 
mixed methods approach to document the dosage and quality of the core components of the 
intervention, and the process and procedures for the delivery of each component. 2 The data on 
the dosage and quality of FOL implementation were drawn from a number of sources: 

• Dosage data were collected on a regular basis throughout the intervention 
year for the training, classroom-level consultation, individualized child-level 
consultation, and stress management components. Dosage was measured by 
attendance logs that documented (1) teacher attendance at the training ses- 
sions; (2) CCC attendance in the classroom; (3) teacher attendance in the 
classroom; (4) Individualized Child Treatment Plans, which included the be- 
havioral goals for children; and (5) One-on-One Reports, which documented 
the clinical interventions that CCCs used in one-on-one sessions with chil- 
dren in order to meet the behavioral goals that were set for the children. 

• Measures of quality were drawn from (1) lead and assistant teacher evalua- 
tions at the end of each training; (2) teacher evaluative reports of the CCCs; 

(3) stress management workshop surveys; (4) focus groups and individual in- 
terviews with lead and assistant teachers; (5) intervention and programmatic 
research notes; and (6) focus groups with the field-based staff in Newark, 
who are described below. Training quality was also assessed by MDRC 


2 See Appendix B for a description of all implementation data sources used in FOL-Newark and FOL- 
Chicago. 
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through observation of each training session and the surveys that were com- 
pleted after each session. In addition to an end-of-the-year survey completed 
by teachers, the quality of classroom-level consultation was assessed by peri- 
odic conversations with teachers, classroom observations by the technical as- 
sistance/research team, 3 and ongoing contact with the CCC supervisor. Stress 
management quality was assessed through an end-of-workshop, close-ended 
survey that teachers completed. 

• Measures that captured the day-to-day operational procedures and that were 
used to implement the intervention were drawn from (1) Service Provision 
Forms (SPFs) that CCCs completed on a weekly basis; (2) research notes 
from weekly supervision meetings with CCCs and the technical assis- 
tance/research team; (3) focus groups with CCCs; and (4) teachers’ perspec- 
tives on FOL implementation as expressed in focus groups, open-ended indi- 
vidual interviews, and surveys. 4 CCCs were the primary reporters of the 
process of FOL implementation, and their process measures (through SPFs) 
included such details as the type of consultation activities that were conduct- 
ed, teachers’ application of key strategies from The Incredible Years within 
the classroom, and teachers’ responsiveness to the overall intervention. 


Implementation Context 

The Newark and Chicago policy contexts that are described in Chapter 1 directly affected the 
structure of the local early childhood education system as well as the resources available at the 
classroom level (for example, funding, staffing, school supplies). The Newark preschool system 
had been infused with substantial resources as a result of the Abbott v. Burke court decisions, 5 
while Chicago was more representative of a typical urban preschool system. 

Newark Classrooms 

The Abbott v. Burke rulings in New Jersey mandated that the State provide free pre- 
school education to all children in certain low-income municipalities whose parents 
wish to enroll them. Additionally, the Court ordered that: 


3 The technical assistance team comprised MDRC staff who were responsible for carrying out the technical 
assistance activities described in Chapter 2. Members of the technical assistance team also served on the im- 
plementation study research team. 

4 A11 focus group data were analyzed in NVivo. Survey data were analyzed in SAS or Excel. 

5 New Jersey Department of Education (2008). 
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• All lead teachers are required to have a minim um of a bachelor’s degree. 6 

• All classrooms must have a lead teacher and an assistant teacher. 

• Each preschool classroom must have no more than 15 students. 

• The size of each preschool classroom must be at least 950 square feet. (Since 
this mandate could necessitate construction or renovation of facilities, pro- 
gram sites with classrooms that were smaller than this size were initially al- 
lowed to remain within the Abbott system.) 

• Each preschool classroom must have a standard set of materials and distinct 
activity areas for children, including a sand and water table, a dramatic play 
area, a library, and an area specifically for playing with blocks. 

• All facilities must provide special education, bilingual education, and health 
services. 

• Teachers must use a standard curriculum assigned by the school district. At 
the time of this study, Newark used the Creative Curriculum. 7 

Despite the infusion of resources into Newark preschool classrooms, initial pilot testing 
of FOL identified a shortfall in resources to address the social and emotional needs of children. 
Newark’s primary mechanisms to address these issues were the Preschool Intervention and Re- 
ferral Team (PIRT), which operates through the public school district, and mental health con- 
sultants (MHCs), who are available to Head Start centers as part of the program’s federal man- 
date. 8 This field-based staff was responsible for conducting observations of children and 
referring children for additional services once a need was identified. Teachers in public schools 
also had “as needed” access to school-based social workers. 

Teachers also had access to resource teachers (that is, mentors) who provided infor- 
mation and support on a wide range of educational issues, such as development of lesson plans, 
classroom set-up, and accessing school-based, center-based, or external resources for children 
or families who were in need of additional support. Importantly, teachers reported that school- 
and center-based referrals for services to address children’s challenging behaviors often took a 
significant amount of time to process in Newark. While the staff was committed to meeting the 


6 This requirement, coupled with a substantial increase in financial resources, made Abbott-funded Newark 
preschool teachers among the highest paid and most credentialed in the nation (Barnett et al., 2008). 

7 See www.creativecurriculum.net for more information on this curriculum. 

8 See Appendix Table B.l for a description of supplemental staff who worked within the early childhood 
education system in Newark and Chicago. 


36 



needs of the highest-risk children, they readily acknowledged that the system could benefit from 
a greater number of adults to attend to children with challenging behaviors. 

Chicago Classrooms 

Although the FOL-Chicago classrooms were located in a range of settings, they were 
all part of the Head Start system. This meant that they were held to Head Start regulations set at 
the national level, which at the time of the demonstration included: 9 

• All lead teachers are required to have a minimum of a Child Development 
Associate degree (CDA). 10 

• All classrooms must have two co-teachers or a lead teacher and a teacher’s 
aide or assistant. 

• Each classroom has a mandated maximum child-teacher ratio of 20:2. 

• Facilities must have 35 square feet of indoor space per child. 

• Classroom materials must be supportive of educational objectives and chil- 
dren’s culture, and must be developmentally appropriate and safe. 

• All centers must have a mental health professional available on a schedule 
that is of sufficient frequency to meet the needs of the children. 

• Developmental screenings must be completed on all children, and families 
should be linked to appropriate resources for special education if necessary. 

Thus, the educational requirements for teachers were lower in Chicago than in Newark, 
while the minimum child-teacher ratio was higher in Chicago. Moreover, in weekly technical 
assistance meetings and a focus group, CCCs reported entering understaffed classrooms in 
which the 20:2 ratio was not met in Chicago because only one pennanent teacher was present. 
Teachers and CCCs recalled situations in which teaching staff were either reassigned to differ- 
ent classrooms in a center or school or were floating between classrooms throughout the day to 
cover staffing shortages. During interviews, CCCs and teachers alike echoed the stress that 
teachers experienced as a result of staffing shortages in Chicago Head Start classrooms and the 
ways in which the stress affected their implementation of FOL, as described by this CCC: 


9 For more information, see “Head Start Program Performance Standards and Other Regulations’’ at 
http://eclkc.ohs.acf.hhs.gov/hslc/standards/Head%20Start%20Requirements. 

"Mead Start was also moving toward requiring bachelor’s degrees for lead Head Start teachers. 
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A lot of my teacher’s stress came from staffing. [There] is always a shortage 
of staff... There’s a change on a daily basis, so it’s very inconsistent. That’s 
always stressful. 

Teachers had access to “master teachers” who served as mentors and to social workers 
employed by the Chicago Public Schools. 11 The FOL-Chicago classrooms also had access to 
MHCs because of their status as Head Start sites. The MHCs were typically employed through 
a contract with an outside agency and were available to provide parent training, teacher training, 
and referrals to mental health professionals or other services that a child or family might need. 
(One center had an on-site MHC who provided therapy to children.) Teachers found the addi- 
tional support services from these professionals to be helpful in infomiing their practice; how- 
ever, they also reported a lack of consistency in the receipt of these services. During the imple- 
mentation year, teachers and CCCs observed master teachers who, instead of providing training 
and coaching to classroom teachers, would at times serve as a classroom teacher themselves to 
compensate for staffing shortages. This limited the master teachers’ ability to provide profes- 
sional development support to classroom teachers, as noted by one teacher: 

I wish she [the master teacher] would be around more often but the reason is 
that they still need a teacher in her [another teacher’s] room so she has not 
been able to co-teach in both rooms the way that we would like it to be.. . . I 
mean, she has not been inside the [my] classroom. We only meet in the 
morning. 


Implementation of the FOL Components 

The dosage and quality were measured for each FOL component (with quality measured by the 
satisfaction of those who were participating in the service). As discussed below, the FOL inter- 
vention was delivered with the intended dosage, and teachers reported a high level of satisfac- 
tion with the training, consultation, and stress management components of the model. However, 
the financial and child care incentives seemed to be more instrumental in getting teachers to 
attend the Chicago trainings (but not necessarily getting them to be more engaged in the training 
activities), and the CCCs in Chicago spent their time in the classroom differently from the way 
the Newark CCCs spent their time. 

Teacher Training 

Attendance at FOL training sessions was consistently high in both Newark and Chicago 
throughout the months of October through January. Table 3.1 shows that in Newark 22 out of 


n See Appendix Table B.l for a description of supplemental staff who worked within the early childhood 
education system. 
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Foundations of Learning Demonstration 
Table 3.1 

Attendance at Teacher Training Sessions, Newark and Chicago 


Attendance 


Newark 



Chicago 


Number of 
Lead 
Teachers 

Number of 
Assistant 
Teachers 

Number of Sites with 
Both Teachers 
Attending 

Number of 
Master or Lead 
Teachers'* 

Number of 

Assistant 

Teachers 

Number of Sites with 
at Least 2 Teachers 
Attending 

Attended 4 to 5 trainings 

22 

19 

17 

20 

15 

14 

Attended 2 to 3 trainings 

3 

4 

5 

4 

6 

3 

Attended fewer than 2 trainings 

1 

3 

4 

1 

5 

3 

Sample size 

26 

26 

26 

25 

26 

20 


SOURCE: MDRC calculations from attendance records for the five Foundations of Learning Demonstration Preschool Teacher and Clinical Classroom 
Consultant Training sessions. 

NOTE: “There were five master teachers included in the sample. One master teacher served as a lead teacher in one classroom. One classroom had two 
lead teachers. 



26 lead teachers received the full dosage of training, defined as attending either four or all five 
sessions. Chicago experienced a similarly high dosage of training, with 20 out of 25 lead and 
master teachers attending four or five sessions. 12 FOL teachers were given a stipend that was 
equivalent to an hourly rate for their teaching positions to attend at each of the training ses- 
sions. 13 The financial incentives for attendance played a part in initially drawing Newark and 
Chicago teachers to the trainings. Particularly in Chicago, teachers reported being under stress 
because of personal financial pressures, and they valued, and in some cases were very much in 
need of, the additional income that the stipends provided. In fact, the stipends seemed to be key 
to increasing attendance at the trainings in Chicago, attracting teachers who might not have at- 
tended because training on a weekend was less than desirable and/or because the stipend met a 
financial need, as this teacher explained: 

I think the stipends really helped.. . . On Saturdays [we] really don’t want to 
do nothing... [and] the stipends [were] really a motivator. And I hope that 
[doesn’t] sound tacky. . . ’cause we are underpaid. . . . 

Another teacher stated, “I’ve been working at the center for 20 years. I’ve got two raises since 
I’ve been there, and that was, like, 50 cents. So, that [stipend] means a lot.. . .” 

The quality of the training was assessed primarily through ratings provided by the lead 
and assistant teachers in exit surveys that were conducted at the end of each session. Teachers 
rated the session on a scale of 1 to 5, with 1 meaning “strongly disagree” and 5 meaning 
“strongly agree.” As shown in Table 3.2, teacher satisfaction with the training was high. The 
Newark teachers’ average ratings of training quality ranged from 4.6 to 4.7 on a scale of 1 to 5. 
Chicago teachers’ average ratings were only slightly lower, ranging from 4.5 to 4.6. On aver- 
age, teachers from both sites were in strong agreement that the training content was clear and 
was delivered in an environment that was conducive to learning, and that the trainings enhanced 
their professional development. 

Although teachers expressed a high level of satisfaction with the training, opinions var- 
ied about the strategies that were presented in the sessions, driven mostly by personal factors 
such as whether certain strategies reflected or conflicted with teachers’ own philosophies about 
classroom management. Some teachers firmly disagreed with a few of the key techniques that 
the trainers presented, and they made decisions not to incorporate those techniques into their 


12 Master teachers were employed by Head Start centers to provide on-site support to preschool teachers on 
a routine basis. 

I3 in Newark, teachers received a $300 stipend to attend each training session, and in Chicago the stipend 
was $225. The stipend amount reflects the variation in teachers’ average salary between the Newark and Chi- 
cago demonstration sites. Teachers who expressed a need also received financial assistance for child care. 
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The Foundations of Learning Demonstration 
Table 3.2 

Quality of Teacher Training, as Rated by Teachers 


Teacher Feedback 

Mean Rating 1 * 
Newark Chicago 

Training content was clear 

4.7 

4.5 

Training environment was conducive to learning 

4.7 

4.6 

Trainers were effective and clear 

4.6 

4.6 

Training enhanced professional development 

4.7 

4.5 

Sample size (teachers) 

52 

51 


SOURCE: MDRC calculations from the Teacher and Clinical Classroom Consultant Training 
evaluation forms. 

NOTES: Responses come from teachers at 5 training sessions. 

“Training sessions were rated on a scale from 1 to 5, with 1 representing "strongly disagree" 
and 5 representing "strongly agree." These ratings represent an average across the five training 
sessions for all participating teachers. 


classroom practices. A lead teacher stated, 

“[T]ime out” is not something we use. . .and maybe it’s just your own values, 
your own beliefs, your own cultural upbringing.... It doesn’t make it [the 
strategies] a bad thing; [they are] something you just disagree with.... Like 
with the “turtle” [technique] 14 . . . I just don’t like that. 

The statement below illustrates how training attendance by both the lead and assistant 
teachers in the classroom provided a vehicle for discussion about classroom management pref- 
erences, allowing teachers who had differing perspectives to discuss their disagreements using a 
common framework: 

I think one of the most helpful things, just in retrospect, is that every person 
at the center came to these trainings, and I think that’s so valuable be- 
cause. . .a lot of this [training content] aligned with my personal philosophies 
on how to interact with children, but. ..every person comes at it from a dif- 
ferent vantage [point].... It’s really helpful to have a bottom line common 


14 The turtle technique is an Incredible Years strategy used to help a child regulate his or her emotions. The 
child is asked to imagine she has a shell and to take three deep breaths to relax as she calmly reminds herself 
that she is in control. 
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ground where you come together. . .[and have] some expectations for how we 
should be interacting with children. 

Teachers also welcomed confinnation that some of what they were already doing in the 
classroom reflected best practices. An assistant teacher stated, 

It [the training] let me know that I was doing some of the things that I should 
have been doing. Just reassurance. ’Cause some of the things that I do, I 
didn’t know why I did it. 

Another teacher agreed: 

Some of the things that I was already implementing, to see that it is a strate- 
gy..^ technique... giving what I was already doing a new name, so to 
speak.. . . I was reassured in what I was doing that, okay, there’s a reason be- 
hind it. . . . 

Given that teachers were able to establish this connection with what they were already 
doing, it was fairly easy for them to incorporate the training techniques into their teaching. 

However, teachers often did not internalize the complete philosophy of The Incredible 
Years curriculum and training content. Interviews with teachers revealed that they regularly 
chose strategies without regard for the fact that The Incredible Years curriculum is based on a 
specific sequencing pattern. 

As one teacher reported, 

No one is going to walk away with the whole book and do everything — but 
the things that actually stood out, the things that I felt as though they were 
important enough to be implemented, I think they will go along whether the 
CCC is there or not. 

When asked whether teachers understood that the intervention was based on a pyramid 
and should be implemented strategically and in a sequenced manner, one CCC responded, 

I ended up revisiting the discipline hierarchy. Because even that discipline 
hierarchy is. . .based on. . .the bigger thing [intervention], and to them [teach- 
ers] it was, like, “Wow!”.... They still got the main concepts, maybe not the 
building on it. 

The training seemed to serve several purposes, including validation of current class- 
room practices, provision of opportunities for co-teachers to collaboratively reflect on and dis- 
cuss classroom procedures, and development of a beginning understanding of The Incredible 
Years curriculum. 
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The classroom consultation component, described in detail below, provided an addi- 
tional opportunity to work with teachers to deepen their understanding of The Incredible Years 
strategies, including how to appropriately apply the strategies within their classrooms in accord- 
ance with the staged and sequenced curriculum tenets. 

Classroom Consultation 

CCCs in Newark provided an average dosage of 183 hours (or about 26 days) of class- 
room consultation services to individual classrooms over the course of the program year. 15 In 
Chicago, the dosage of classroom-level consultation was higher, at 217 hours (31 days). Imple- 
mentation in Chicago drew on lessons from the Newark experience, which was expected to fa- 
cilitate a higher dosage of classroom-level consultation hours. For example, the CCCs in Chica- 
go were hired sooner and, unlike in Newark, there was no turnover in staff. 16 Chicago 
consultants also received explicit guidance that vacations could be taken only during times 
when programs were not in session. These differences account for some, but not all, of the vari- 
ation in consultation hours. 

While the overall dosage of consultation that teachers received was high in Chicago, an 
examination of what actually occurred when CCCs and teachers spent their time together is crit- 
ical in order to understand how CCCs worked with teachers to help them improve their effec- 
tiveness. The consultation model stipulated that CCCs focus their initial work with teachers on 
establishing positive relationships, which also included time spent clarifying roles and responsi- 
bilities. Despite the consultant orientation and written guidelines about this part of the work, 
teachers, administrators, and CCCs in both Newark and Chicago struggled to understand how 
and in what capacity CCCs could be utilized. As one CCC explained, 

It was kind of challenging. . .trying to explain your role.. .. At the beginning, 
it [CCC role] was really hazy, even for myself. .. [it] felt really abstract. I 
didn’t know really what that meant until... we went to the training. 

And. ..that solidified everything for us.... Then we needed to revisit... what 
my role was in the classroom. ... It had to be done several times and some- 
times even with the administration. . . . 

This role clarification appeared to be important to set expectations for how teachers and 
CCCs should engage with each other. The quotation above also suggests that roles may need to 


15 In a previous report (Lloyd and Bangser, 2009), the average dosage of consultation services in Newark 
amounted to 162 hours (23 days) when calculated based on an average of the hours spent with lead and assis- 
tant teachers. The program anticipated that there could be as many as 36 days of classroom consultation over 
the course of the year. 

16 Two of the CCCs in Newark were hired late and two were replaced. 
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be reviewed over the course of the year with multiple stakeholders, including administrators, 
who are important to the intervention implementation process. 

In addition to role clarification, more generalized relationship-building activities, such 
as assisting teachers and children in the classroom, were expected to continue throughout the 
program year. However, activities that were directly related to the implementation of The In- 
credible Years curriculum were also expected to occur routinely as part of classroom visits be- 
ginning around the end of October (after the first training session). Data from the Service Provi- 
sion Fonns, submitted by CCCs after each classroom visit, were analyzed to understand the 
content of the CCCs’ sessions with teachers. 

Table 3.3 shows that relationship-building activities did indeed take place throughout 
the year (approximately 26 weeks) in both Newark and Chicago. In Newark, substantial atten- 
tion was also given to the implementation of The Incredible Years, including reviewing training 
material with teachers, and a sustained focus on the classroom-based intervention. CCCs in 
Newark spent 15 weeks consulting with teachers on strategies from The Incredible Years cur- 

The Foundations of Learning Demonstration 
Table 3.3 


Average Number of Weeks and Percentage of Total Weeks 
Spent on Each Consultation Activity 


Type of Consultation Activity Reported 

Newark 

Average Percentage 
Number of of Total 

Weeks Weeks 

Chicago 

Average Percentage 
Number of of Total 

Weeks Weeks 

Assisting activities 3 

26.5 

73.5 

26.8 

74.4 

Coaching teachers on The Incredible Years 

15.1 

41.9 

7.2 

19.9 

Reviewing training materials with teachers 

14.8 

41.2 

8.2 

22.8 

Classroom-based intervention 

13.3 

37.1 

6.6 

18.3 

Teacher stress reduction 

5.1 

14.1 

4.1 

11.4 

Collaboration with other field-based professionals 11 

3.2 

8.9 

0.9 

2.5 

Direct child-level service provision 

3.0 

8.3 

5.1 

14.2 

Consulting with parents/guardians 

2.5 

7.1 

1.1 

2.9 

Securing parental consent for one-on-one intervention 

1.9 

5.3 

1.7 

4.6 

Referral/providing resources to parents 

0.5 

1.4 

0.3 

0.7 

Total number of weeks of consultation 0 

36 
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SOURCE: MDRC calculations from weekly Service Provision Form (SPF). 

NOTES: "Referral/providing resources to teachers" and "Collaboration with center staff (for example, 
director, cook)" were two activities on the SPF that did not occur, according to the responses given. 

“These activities include assisting with morning activities, meals, recess, classroom clean-up, and field trips. 
b On-site professionals may include the Preschool Intervention and Referral Team, mental health consultants, 
social workers, nurses, and resource teachers. 

“Actual number of program weeks may vary by site since programs started at different times and site 
closures (holidays, professional development) were not consistent across sites. 
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riculum and 1 5 weeks reviewing the content from training. By contrast, CCCs in Chicago spent 
only 7 weeks consulting on The Incredible Years strategies and 8 weeks reviewing training con- 
tent, or roughly half the time that CCCs in Newark spent on these consultation activities, but 
they devoted significant time to routine “assisting activities,” as did the CCCs in Newark. 

Figures 3.1 and 3.2 show the percentage of classrooms in Newark and Chicago, respec- 
tively, that are engaged in three general categories of consultation activities during each month 
of the school year: (1) activities related to assisting teachers with routine tasks in the classroom; 
(2) activities directly related to implementation of strategies from The Incredible Years curricu- 
lum; and (3) activities to increase organizational support for the FOL intervention, such as meet- 
ing with administrators, mental health consultants, and other key early childhood staff. Figure 
3.1 indicates that Newark CCCs generally adhered to the intervention timeline that was illus- 
trated in Chapter 2 (Figure 2.1) and articulated in the manual and training. In Chicago, fewer 
classrooms engaged in activities that were directly related to implementation of The Incredible 
Years program immediately following the first training, as well as for the remainder of the pro- 
gram year, as depicted in Figure 3.2. 

The analyses of SPFs also indicate that CCCs in Newark spent more time collaborating 
with other field-based professionals (as shown in Table 3.3 and Figure 3.1). Given the greater 
presence and frequency of professional supports that were provided to Newark classrooms, 
CCCs may have simply had more opportunities to interact with these professionals. Moreover, 
regularly scheduled meetings with the field-based staff in Newark, such as the PIRT team and 
resource teachers, 17 led to the development of relationships and shared knowledge about each 
other’s roles and coordination around children’s needs that did not occur in Chicago, despite 
multiple efforts by the technical assistance team to coordinate these activities. 

Thus, the emphasis and timing of consultation activities in Chicago shifted from what 
the training and manual originally advised. Several factors affected the CCCs’ and teachers’ ca- 
pacity to engage in the classroom management strategies. As noted earlier, teachers in Chicago 
were often stretched thin because of staffing shortages in their classrooms. This constraint un- 
doubtedly limited the ability of teachers and CCCs to focus on implementing strategies from the 
FOL training in those Chicago classrooms that were understaffed, as illustrated by the comments 
of one Chicago teacher: “We are so understaffed, with really difficult children... I can’t tell 
whether or not. . .anyone’s trying to do the best practices. . ..Everyone’s. . .at their wit’s end.” 

Not only did the lack of staffing impede teachers’ ability to take on the intervention in- 
depth, it also had the potential to derail the primary focus of the CCCs’ work in the classroom, 


17 See Appendix Table B. 1 for a description of the supplemental staff who worked with the early childhood 
system. 
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Percentage of classrooms 


The Foundations of Learning Demonstration 

Figure 3.1 

Percentage of Newark Classrooms Engaged in Each Category of Consultation Activities 
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The Foundations of Learning Demonstration 
Figure 3.2 

Percentage of Chicago Classrooms Engaged in Each Category of Consultation Activities 
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as suggested by this teacher’s comment: “I know I enjoy having her there, just to have an extra 
body, and extra set of eyes. ... So we actually need her to be there to help us.” This teacher was 
quite fond of the CCC who was assigned to her classroom; however, her comment indicates that 
she viewed the CCC primarily as a helper, or someone to pick up the slack in the day-to-day 
program operations, rather than as someone with whom to engage in professional interactions 
and active consultation. 

Lead teachers also completed surveys that contained specific questions about the ways 
that the CCCs helped them implement what they learned from the training sessions. Table 3.4 
shows that teachers in both Newark and Chicago rated their CCCs as “very effective” in helping 
with classroom management, building relationships with children, and spending more time 
teaching despite the differences between the two demonstrations in the types of consultation 
activities that CCCs reported. Although the ratings were still relatively high for managing rela- 
tionships with their coworkers and with parents, and managing personal issues such as stress, 
teachers from both demonstrations tended to rate their CCCs as less effective in these categories. 

Given that the focus of the FOL intervention, including the teacher training sessions, 
was on teacher-child interactions, it is not surprising that teachers rated their CCCs as most ef- 
fective in domains related to their work with the children. However, even though Chicago 
CCCs engaged teachers in the implementation of classroom management techniques less fre- 
quently than did the CCCs in Newark, both sets of FOL teachers rated their CCCs as “very ef- 
fective” in this area. 

These findings raise an important question: did the variable amount of support from 
CCCs generate the amount of teacher change that is necessary to exert a positive impact on stu- 
dent behavior, and if yes, how? To some extent, this question is answered in Chapter 4. Under- 
standing the second half of this question about mechanisms of change and how the model works 
is a more complex issue and one that can only be discussed theoretically at this point. 

The implementation data suggest that two types of teachers might be more apt to bene- 
fit from the FOL intervention. One group, teachers who were new to the field and/or engaged in 
educational pursuits, may have been more open to taking on the intervention. A comment by a 
Chicago-based CCC sheds light on this consideration: 

The ones who were more open were either. . .in school part time or kind of 
fresh in the field. Whereas the ones who were more set in their ways, you 
know, had been there for a long time. . .and might not necessarily have the 
level of education that some of the others had. . . . Something about that 
school thing. ..somehow, they had a way to relate to the material; [they] 
still had that energy to want to try different things. 
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The Foundations of Learning Demonstration 
Table 3.4 

Teacher Evaluation of the Clinical Classroom Consultants 


Teacher Feedback 

Mean Rating 
Newark Chicago 

How effective was the Clinical Classroom Consultant in helping me: 
Work on classroom management techniques 

3.6 

3.8 

Work on building your relationships with children 

3.8 

3.9 

Manage relationships with parents and coworkers 

3.1 

3.5 

Manage personal issues, like stress 

3.0 

3.3 

Spend more time teaching 

3.8 

3.8 

Sample size 

23 

16 


SOURCE: MDRC calculations of teacher responses to Clinical Classroom Consultant evaluation form. 

NOTE: Clinical Classroom Consultants' effectiveness was rated on a scale from 1 to 4, with 1 representing 
"not at all effective" and 4 representing "very effective." 


A second group of teachers, those who were struggling and who had a clear and 
immediate need for the intervention, might also be particularly responsive to consultation. 
One CCC stated: 

I noticed. . .the teachers that were more open were more desperate... they 
were like, “Help me. Somebody, help me. ..I’m gonna do everything you 
say, if it doesn’t work, it doesn’t.”... One teacher in particular... I [also] 
had two more . . . from day one to the end, she stayed at the same level of 
[high] engagement. . . . She would change from not having a [co-]teacher, 
having a [co-]teacher. . .having a problem child, not having him. [The 
year] was very inconsistent for her and. . .that level was still there because 
she just was at a loss.... Desperation played a part in how open some of 
those teachers were. 

In addition, the administrative staff at the preschool sites had a role in supporting teach- 
ers’ capacity to engage in the classroom management strategies. Responses from CCCs and 
teachers during qualitative interviews described a range of characterizations of the administra- 
tion, from offering overt support for teachers, to having little or no involvement, to sabotaging 
teachers’ efforts to practice the classroom management strategies. Administrators who had a 
clear sense of the intervention and the CCCs’ purpose tended to share information about the 
intervention with teachers at the beginning of the year, which appeared to support an initial level 
of buy-in among the classroom teachers: 
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Upon being hired, my director explained to me that the classroom that I was 
going in would be participating in the study.... She explained what the re- 
search was going to be about. . .and I was excited. 

Administrative staff who lacked a full understanding of the intervention in some cases 
hindered teachers’ capacity to engage in the classroom management practices. For instance, a 
program director or school principal may undermine a teacher’s application of a particular strat- 
egy by pulling a child into his or her office as opposed to allowing the teacher to consistently 
work on ignoring certain negative behaviors, which is an Incredible Years strategy. A CCC de- 
scribed how an administrator’s actions could potentially derail not only the effectiveness of the 
strategy, but also the teacher’s motivation to engage in the intervention: 

At one point, after a while [he] just kind of gave up and went back to where 
he was originally and said, “Well, you know, I’m not going to fight admin- 
istration, I’ve got to work with them here. . . . I’ve got to do what they’re ask- 
ing me to do.” So, when I was there, you would see him trying to do it, and 
then of course all the other days he was just trying to do what administration 
was trying to do. 

Teachers’ openness to consultation was, however, only one part of the equation. Prepar- 
ing CCCs for their role might also be important. In particular, CCCs struggled to provide feed- 
back that was robust enough to change teacher practices, and they cited fear of damaged con- 
sultant-teacher relationships as a barrier to this process. Indeed, teachers often said they were 
initially leery about the presence of a CCC in their classroom as an intrusion and worried that 
CCCs might “spy” on them and report back to administrators. 

One CCC reflected the sentiments of both the FOL-Newark and FOL-Chicago CCCs: 

I had meetings with all the teachers once a week with those debriefings and I 
would push myself to bring up things. But it took me a long time to be com- 
fortable to address an issue, like, while it’s happening, or confront the teach- 
er, you know?. . . I didn’t want to push them or make them feel like they were 
being pushed. So it took me a while, probably more than halfway into it [the 
academic year], to feel comfortable to bring something up. 

Some teachers, however, wanted more constructive feedback: 

I really would have liked her to talk about how I could improve or how I was 
actually doing things. . .because it was more of, like, I’m here for you, what- 
ever you need, but that often turned into, like, sweeping the floor. 

When feedback was provided skillfully, consultant-teacher relationships were not 
threatened, and the conversations created a forum for teachers to reflect upon and improve their 
practices. The process, however, was not as straightforward as was designed. CCCs were ex- 
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pected to engage in weekly meetings with both the lead and assistant teacher present to reflect 
on intervention implementation processes and to set goals for the following week. The quota- 
tion below and data from the SPFs suggest that it might not be realistic to expect CCCs to find 
separate dedicated time for reflection, debriefing, and goal setting with both teachers. Instead, 
feedback might need to be provided in conjunction with other, more routine consultation activi- 
ties. A lead teacher stated: 

Our consultant was... very good... very helpful.... She did give us feedback, 
like around the time the kids [were] laying down, and she would 
say. . .“[What] did you think about trying... the different strategies that we 
had tried to implement?” She was, like, did they work, did it not work?... 

Are you going to do it for the rest of the week or, you know, things like 
that.. . . At the same time she was very helpful within the classroom, too.. . . It 
is good that they’re able to multitask, because at the same time of helping 
you, they’re still, you know, observing... and noticing things that’s working 
or. . .that may not be working.... She instigated a lot of reflection. 

Helping CCCs feel empowered enough to address these issues with teachers required a 
strong supervision component. In Newark, technical assistance meetings with CCCs and the 
CCC supervisor occurred in person, with a member of the technical assistance team in attend- 
ance to lead the discussion. In addition, the technical assistance team completed periodic check- 
ins and coordinated regularly with the clinical supervisors. The in-person meetings, coordina- 
tion with the clinical supervisor, and the consistent presence of the technical assistance team in 
the preschool sites allowed for direct communication among the technical assistance team, the 
CCCs, and CCC supervisors; this provided a clear understanding of on-the-ground implementa- 
tion of the classroom-level consultation model and promoted more responsiveness to emerging 
challenges and support for CCCs. In Chicago, however, the technical assistance meetings 
proved more difficult to coordinate from a distance. The technical assistance team was based in 
New York, so face-face-meetings were held less often and most communication occurred by 
phone. This long-distance coordination of technical assistance for the consultation staff may 
have been a factor in the CCCs’ reduced focus on initiating classroom management consultation 
in the Chicago classrooms. 

Individualized Child-Centered Consultation 

The CCCs provided one-on-one services to a total of 63 children in Newark and 58 
children in Chicago. The number of children receiving one-on-one services ranged from zero to 
five per classroom in Newark. In Chicago, two to four children per classroom received these 
services. Over the course of three months, children in Newark received an average of 4.7 one- 
on-one sessions, and in Chicago, children received an average of 7.8 sessions. In Newark, 
CCCs encountered delays in the initiation of one-on-one services because of difficulties secur- 
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ing parental consent. After learning from the implementation experience in Newark, CCCs in 
Chicago were asked to begin seeking parental consent for individualized services early in the 
school year, which likely facilitated the higher dosage. In addition, CCCs in Newark were re- 
quired to be licensed to provide clinical services, and delays in licensure led to fewer one-on- 
one sessions. 

By design, the one-on-one services did not follow a standardized format, leaving CCCs 
with the autonomy to conduct sessions and develop individualized treatment plans for children 
based on their own clinical knowledge and skills. They did, however, receive support for and 
review of this work from their clinical supervisors. Because the sessions were largely individu- 
alized for each child, the implementation analysis does not provide information about the spe- 
cific types or the variation of the one-on-one activities that occurred, leading to less clarity about 
the effects of this work in causing behavior change among children. 

Despite the more fluid structure, the prospect of the one-on-one sessions had been a ma- 
jor factor in gaining teachers’ initial buy-in to FOL. Even before the services were initiated, 
teachers expressed enthusiasm about the possibility of clinical services for children, the over- 
arching sentiment being that it was one less concern that they would need to address. One 
teacher in Chicago shared her impression that the availability of one-on-one services reduced 
her initial hesitancy about FOL: 

I just thought it [FOL] was. ..something extra that I probably didn’t want to 
be a part of. But the high points of the project that did strike a note in me 
were the fact that they were going to provide the clinical consult- 
ant. . .because I felt that someone who’s trained to deal with children with be- 
havioral issues or problems that maybe we and my staff [were] not. . .as able 
to contend with. . .if they had another person there who may be able to offer 
some strategies or ideas on how to deal with those type[s] of problems with 
children, that would be beneficial. 

Teachers felt that one-on-one sessions benefited their classrooms and the individual 
children who received the services. The sessions relieved teachers so they could focus on 
instruction for the entire class, without the need to respond to disruptions caused by chal- 
lenging behaviors: 

P]t’s like you’re leaving a smoke-filled room, walking out into a breath of 
fresh air.. . .You have this one person who comes in and lets the child know 
it’s all about you.... We [teachers] never say, “Yeah, it’s all about 
you”. . .because you have to give a piece of yourself to everybody that’s in 
the classroom. 

Teachers reported a high level of satisfaction with the children’s response to the one-on- 
one sessions, citing examples during the interviews and focus groups of the changes they ob- 
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served in children’s behaviors. The one-on-one services not only improved their perception of 
children’s behaviors, but also helped to reframe teachers’ understanding of children who were 
exhibiting difficult behaviors. As described by teachers, this new lens eventually led to better 
understanding about the cause of the behaviors, and in some cases increased the teachers’ sensi- 
tivity toward the children and decreased the tendency to label them as a problem. One teacher 
who had experienced significant anxiety when engaging with a particular child eventually felt 
differently, as she describes here: 

This kid did not have any friends. He did not have friends in this class be- 
cause he would hit. . . . No one wanted to get close to him. Well, thanks to the 
planning that the [CCC] helps me with. . .to help that specific kid. . .that kid 
today. . .plays with everyone. ... I feel so happy to see that kid now. 

Moreover, teachers’ felt that the one-on-one support and the resulting decrease in class- 
room disruptions enabled them to focus on supporting the whole class, providing space in the 
day for learning opportunities that might have otherwise been lost. 

I[’ve] seen the change in him and I[’ve] seen that the program is working for 
him, for a lot of kids. And now I can just sit in the room with them... . I can 
really teach the children. . .and I don’t have to chase them all over the room 
[anyjmore. 

Stress Management 

The final component of the FOL intervention was stress management. While this was a 
small component of the overall intervention, it resonated universally with teachers in both cities, 
but especially in Chicago, likely in part because of the challenges with classroom staffing and 
resources there. Teacher stress resulted from a number of factors, including the demands of their 
job, lack of consistent staffing and resources necessary to carry out their lessons and activities, 
children exhibiting difficult behaviors, and teachers feeling overwhelmed by their work respon- 
sibilities. In addition, teachers felt that stress from their personal lives at times carried over into 
their work. 

FOL addressed teachers’ stress in multiple ways: providing concrete classroom man- 
agement strategies during training, as well as the consultants’ reinforcement of those strategies 
in the classroom; the provision of individualized, one-on-one interventions for children who 
exhibited challenging behaviors; stress management workshops; and ongoing support from 
CCCs about stress-related issues throughout the year. Teachers reflected on the ways in which 
their participation in FOL effectively decreased the stress they felt in their classroom by giving 
them the training and tools they need to manage behaviors: 
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It really calmed me down because I didn’t know what to do.. . . Now I know 
that I can use words to praise my children, ignore the children, and I don’t get 
frustrated [any] more. I don’t [get] stressed out.... I [don’t] wanna go home. 

In addition, the reduction of acting-out behavior, which some teachers saw as a direct 
result of the CCCs’ consultation activities with them and with the children, helped to relieve 
some of the stress they felt in response to the difficult behaviors: 

Well, I was a little under stress.. . .That’s really important not to be stressed at 
home and then go to work and be stressed out. . . . When you’ve got this child 
jumping and flipping all over... once she stopped... once she slowed down, 
it’s like the whole entire classroom [calmed down]. So I tip my hat to this 
program.... 

Finally, FOL included a stress management workshop to provide the teachers with 
techniques for developing an awareness of their own stress levels and ideas for addressing the 
source of their stress, whether it is personal or work-related. Attendance at stress management 
workshops was generally high in both demonstrations, and teachers expressed a significant level 
of satisfaction with this component. Forty-nine out of 52 lead and assistant teachers in Newark 
and 39 out of 51 lead teachers, assistant teachers, and master teachers in Chicago attended a 
workshop and completed an evaluation of it. Attendance was facilitated by providing the work- 
shops at the teachers’ schools or centers. In addition, the content for the workshops was individ- 
ualized based on the CCCs’ knowledge of teachers’ personal and professional stressors. The 
topics that were discussed during the workshops included managing deadlines, dealing with 
difficult colleagues, lack of personal autonomy and self-efficacy at work, domestic violence, 
financial challenges, physical and mental health, and managing familial responsibilities. 

Teachers were asked to complete an evaluation at the end of each workshop. Evaluation 
criteria included teachers’ understanding of stress and what impact it had on them professional- 
ly, their ability to identify the sources and symptoms of stress, and their understanding of how 
they could manage stress. Finally, teachers were asked whether the workshop was useful in 
helping them to manage and/or reduce their stress levels. 

Table 3.5 shows that lead and assistant teachers in FOL rated the workshops and the fa- 
cilitators very positively, with Newark teachers rating them slightly higher. Teachers found the 
stress management workshop helpful because it provided them with an opportunity to express 
the difficulties they were experiencing within their professional environment and personal lives 
and to reflect on how much their stress contributed to challenges in the classroom. In both cities, 
teachers found the workshops to be valuable enough to recommend that the intervention incor- 
porate additional and longer stress management workshops. 
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The Foundations of Learning Demonstration 
Table 3.5 


Stress Workshop Evaluation by Teachers 


Teacher Feedback 

Mean Rating 
Newark Chicago 

Overall rating of instruction 2 



Content and knowledge 

4.85 

4.87 

Presentation and style 

4.83 

4.85 

Training ability 

4.87 

4.82 

Teacher self-assessment 15 



I have learned and can implement concrete tools/skills to help me cope with stress 

4.75 

4.64 

This workshop was useful in helping me manage/reduce my stress 

4.83 

4.59 

Sample size 

24 

20 


SOURCE: Responses from teacher survey during Stress Workshop in Newark and Chicago. 

NOTES: “Average score for all instructors, based on a range from 1 = poor to 5 = excellent. 

b Average score for all instructors, based on a range from 1 = strongly disagree to 5 = strongly agree. 


Overall, the implementation analyses indicate that the FOL intervention components were de- 
livered at relatively high levels of dosage and quality. The analysis of process data from consul- 
tation sessions as well as qualitative data from teachers and CCCs also point to several imple- 
mentation challenges. The low level of program resources, especially staffing, presented 
barriers to implementing the intervention in Chicago. Delivery of the classroom consultation 
component appeared to be affected by staffing shortages and teacher receptivity more than any 
of the other components of the intervention. As is discussed in Chapter 6, other factors such as 
the CCCs’ willingness to initiate feedback and differences in supervision of the CCCs between 
the two cities also likely affected the delivery of consultation services. Furthennore, adjustments 
to the dosage of the consultation model may be warranted given the pattern of impacts on 
teachers and children in the FOL-Newark and FOL-Chicago demonstrations. 
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Chapter 4 


The Impact of Foundations of Learning on Classrooms 

and Children 


The Foundations of Learning (FOL) impact evaluation addressed four questions: (1) whether 
teachers who received FOL training and in-class consultation structured more emotionally 
positive, behaviorally supportive classroom environments than similar teachers in the control 
group did; (2) whether the FOL teachers demonstrated higher-quality instructional practices, 
something that was not specifically targeted in the training or in-class consultation; (3) whether 
preschool children were positively affected by the implementation of the FOL model in their 
classrooms and, if so, in what developmental domains; and (4) whether any changes that were 
observed in children’s skills during the preschool year were sustained when the children moved 
on to elementary school. 

As discussed in Chapter 1, these findings are based on the results of three rigorous stud- 
ies — CSRP, FOL-Newark, and FOL-Chicago. In all three studies, outcomes for children were 
assessed in preschool and, to a lesser extent, as they moved into elementary school. It is unusual 
to be testing the same program model successively in three separate studies. Similar results 
across sites would allow substantially stronger statements to be made about the generalizability 
of effects. Alternatively, divergent impacts across sites could provide a basis for developing 
hypotheses about how the program model interacts with the preschool context. 


Summary of Findings 

• Based on observers’ ratings, teachers in the program groups had more 
warm and positive interactions and fewer sarcastic and angry interac- 
tion, were more sensitive, and were better able to prevent misbehavior 
than teachers in control group classrooms; increased instructional time 
was also observed in the program group classes. The first and most fun- 
damental question was whether the intervention had an impact on the strate- 
gies in which teachers were trained, and, indeed, relatively large improve- 
ments were observed in the two sites where this was examined. Moreover, in 
Newark, the intervention was found to improve the teachers’ management of 
classroom time and the time spent on instruction, suggesting that FOL in- 
creased children’s opportunities for learning. 

• For children, FOL led to reductions in problem behavior between peers 
and improvements in children’s approaches to learning (children’s abil- 
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ity to engage in the learning tasks of preschool); however, no clear evi- 
dence of benefits to children’s academic skills was demonstrated. While 
moderate-sized benefits were observed in child outcomes targeted by the 
program (problem behavior and approaches to learning), there were no 
clear benefits for child outcomes that the program addressed less directly 
(that is, secondary outcomes) — children’s social skills (how they get along 
with other children) and early academic skills that will later support reading 
and math. This latter finding shows that benefits to children’s social and 
emotional development may not have the kind of spillover effects in chil- 
dren’s academic achievement that were thought possible at the outset of the 
demonstration. 

• With regard to longer-term outcomes for children, the limited data that 
were collected show no evidence of sustained benefits for children as 
they make the transition into elementary school. Teacher reports and 
school records of children’s academic achievement show no impacts of FOL 
in kindergarten and first grade across the three tests of the intervention, al- 
though rates of grade repetition and special education are too low in these el- 
ementary school years to detect impacts. 

The rest of this chapter presents a more detailed discussion of these findings on class- 
rooms and children during preschool. The findings are presented for CSRP, FOL-Newark, and 
FOL-Chicago, in that order; they are then summarized across sites to identify patterns of 
impacts. The final section of the chapter presents the information that was collected as the 
children moved into elementary school, setting the stage for the discussion of costs and potential 
benefits in Chapter 5. 


Study Design 

As discussed in Chapter 1, in all three demonstrations, preschool centers were assigned, using a 
lottery-like process, to a program group that received the CSRP or FOL intervention, or to a 
control group that provided services as usual. As such, these studies provide information about 
the added effects — or “value added” — of the FOL intervention to preschool services in these 
communities. The sample size was largest in the Newark demonstration, and rich baseline 
information that was collected in the CSRP study increased the power of that study to detect 
program impacts; the power to detect impacts was lowest in the Chicago-FOL site. 

Based on funding constraints, the team decided to focus on collecting complementary 
rather than identical information across the two FOL sites, so that infonnation from one site 
could build on infonnation from the other (and vice versa) and on the earlier findings from 
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CSRP. More specifically, FOL-Newark focused primarily on outcomes that occurred in the 
preschool classrooms (among teachers and children), while FOL-Chicago focused on outcomes 
for children that could be linked to longer-term outcomes in elementary school. In both cases, 
infomiation was collected from sources other than the teachers themselves. Since the teachers 
were the target of the intervention program, they might perceive changes in themselves or in the 
children that would not be reflected in more objective assessments by trained observers who are 
blind to the intervention status. Thus, these more objective measures are critical to the study of 
the impact of the intervention. A summary of the in formation that was collected across the three 
sites is presented in Table 4.1. 

CSRP collected the most extensive infonnation on classrooms and children. It was de- 
signed as a rich efficacy study of the impacts of the model on teachers’ behavior that might 
represent key pathways of influence of the program. In addition, CSRP was designed to collect 
infomiation on outcomes for children who were considered direct targets of the intervention 
(children’s behavior problems and approaches to learning) as well as more distal, or secondary, 
child outcomes, such as children’s early academic skills. This infonnation was collected from a 
variety of sources, including trained observers, trained interviewers, and teachers themselves. 

In FOL-Newark, the priority was to test whether impacts on the most direct targets of 
the intervention — teachers’ ability to provide a supportive emotional climate and manage 


The Foundations of Learning Demonstration 
Table 4.1 

Measures Used in CSRP, FOL-Newark, and FOL-Chicago 


Measures Used 

CSRP a 

FOL- 

Newark 

FOL- 

Chicago 

Classroom outcomes 

Observer rating 

X 

X 


Preschool child outcomes 

Observer ratings of children's behavior in classroom 

X 

X 


Teacher ratings of classroom behavior and academic skills 

X 

X 

X 

Direct assessment of approaches to learning 

X 


X 

Direct assessments of early academic skills 

X 


X 

Kindergarten and first-grade child outcomes 

Teacher ratings of classroom behavior and academic skills 

X 

X 


School records 

X 


X 


NOTE: a CSRP was formerly known as the Chicago School Readiness Project. 
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children’s behavior — would be similar to those in CSRP (which took place in Chicago), 
given the substantially different base of resources required by the Abbott court decision in 
New Jersey. A secondary question that was addressed in FOL-Newark was whether there 
would be “spillover” in secondary classroom outcomes, such as teachers’ language instruction 
and amount of instructional time. Rather than entirely forgoing any understanding of impacts 
on children, trained observers were sent to classrooms to rate children’s behavior with peers, 
teachers, and activities to determine whether training teachers might also lead to changes in 
outcomes for children in the classroom context. It is less expensive to collect information on 
these outcomes than to conduct direct child assessments; however, while these outcomes are 
conceptually linked in the theory of change, they have not been used extensively in the 
preschool intervention literature. 

In FOL-Chicago, the priority was to focus on those outcomes that the literature suggests 
are most strongly linked to outcomes later in middle childhood and adolescence. That focus led 
to the collection of information through direct assessments, with a particular focus on early 
academic skills (which have been linked most strongly in the literature to later outcomes) as 
well as children’s approaches to learning, a direct target of this intervention. Outcomes for 
teachers and classrooms, while critical to the test of the FOL theory of change, cannot be linked 
to long-term benefits, and therefore were not a focus of the data collection in FOL-Chicago. 

In the quantitative impact analyses presented in this chapter, the means of key outcomes 
are compared for program and control group classrooms and children. For all outcomes, the 
analyses account for the grouping of children within classrooms. The model includes selected 
baseline characteristics at each level, intended to increase the precision of the impact estimates 
that are presented. Because random assignment occurred within prespecified “blocks,” that 
block membership is included in all impact analyses. 


Key Dimensions of Children’s Social Behavior and Approaches 
to Learning 

As discussed briefly in Chapter 1, the intervention was designed to promote children’s social 
and emotional competence through the training of preschool teachers to proactively support 
children’s positive behavior and effectively limit children’s problem behaviors. In this report, 
the tenn “competence” is used to indicate the child’s capacity to do something and “behavior” 
is the manifestation of those competencies in everyday life. Social and emotional competence is 
thought to underlie children’s behaviors, particularly in two critical areas that are central to 
longer-term success in school: (1) social behaviors, or children’s positive interactions with peers 
and teachers; and (2) approaches to learning, or children’s ability to focus their attention and 
behavior during classroom activities. 
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Figure 4.1 depicts child competencies — emotional competence, social competence, 
and executive functioning — and the smaller discrete skills that each one comprises, such as the 
capacity to control negative emotions or aggression (aggression control) in the social compe- 
tence domain. Possessing competence is generally a precursor to being able to display closely 
related behaviors, which are shown in the right-hand column of the figure. 


The Foundations of Learning Demonstration 
Figure 4.1 

Heuristic Model of Children’s Social and Emotional Competence and Executive Functioning 

Skills Competence Behavior 
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Notably, the training that is provided by The Incredible Years program targets teachers’ 
support of some of these underlying skills more so than others. In particular, improvements in 
classroom management should be most strongly associated with children’s emotional self- 
regulation in the emotional competence domain and children’s aggression control and behavior 
regulation in the social competence domain. Other early childhood programs that focus on 
social and emotional skills (for example, the Preschool PATHS program) 1 emphasize different 
skills, such as “emotion knowledge” and social problem-solving skills, in the emotional and 
social competence domains, respectively. 

Strongly related to emotional and social competence and associated behavioral corre- 
lates is executive function, shown as a third competency in Figure 4.1. Its underlying skills are 
inhibitory control (children’s ability to control their immediate or automatic response in favor of 
a planned response), flexibility of attention, and working (or short-term) memory. In addition to 
being directly associated with learning outcomes such as achievement, executive function is 
thought to be strongly associated with children’s approaches to learning. 2 As with social and 
emotional competence, The Incredible Years program is thought to support some of the skills 
that underlie executive function more than others, with the strongest effects expected on 
children’s attention and inhibitory control as compared with their working memory. 3 

While all three studies collected somewhat different information on outcomes for chil- 
dren, all of them assessed the two core behavioral outcomes of the intervention in some fashion. 
That is, all collected some information on children’s social behaviors and their approaches to 
learning through teacher reports as well as through more objective observations and assessments. 


Findings from CSRP 

As discussed in Chapter 1, the FOL intervention model drew heavily on the lessons from its 
precursor, CSRP, in which trained observers had rated program and control classrooms using 
four dimensions of the highly regarded measure of classroom quality, the Classroom Assess- 
ment Scoring System (CLASS). 4 Teachers in the CSRP program group classrooms had more 
positive interactions and fewer sarcastic and angry interactions and were more sensitive than 
teachers in control classrooms when observed in the spring of the academic year. Teachers in 
program group classrooms were also more likely to demonstrate improved classroom manage- 
ment practices, showing better skills in monitoring and preventing children’s misbehavior in 
proactive ways. 


Domitrovich, Cortes, and Greenberg (2007). 

2 Blair and Razza (2007). 

3 Webster-Stratton, Reid, and Hammond (2001). 

4 La Paro, Pianta, and Stuhlman (2004); Pianta, La Paro, and Hamre (2006). 


62 



Likely because of these changes in classroom climate, impacts on outcomes for children 
were observed as well. CSRP reduced preschoolers’ internalizing (sadness and withdrawal) and 
externalizing (aggression and defiance) behavior problems, as reported by teachers. 5 Similar 
effects were noted based on independent classroom observations; there were fewer observed 
instances of externalizing, disruptive behavior (for example, physical and verbal aggression) 
among children in program group classrooms. On outcomes assessed by trained interviewers, 
children in the program group demonstrated significantly higher executive function and early 
academic skills than did their control group counterparts. 6 Similarly, despite these benefits 
during the preschool intervention year, there were no sustained effects in CSRP, on average, for 
children’s skills. 7 


Findings from Foundations of Learning in Newark 

As presented in a previous report, 8 FOL-Newark addressed whether the intervention made a 
difference in classrooms, with a focus on teachers’ behavior in multiple domains of practice as 
well as in children’s behavior. Impacts of the FOL-Newark intervention are discussed below. 
(See Box 4.1 for an explanation of how to interpret the impact tables in this chapter and Box 4.2 
for an explanation of effect sizes.) 

Impacts on Classrooms and Teachers’ Behavior 

As in CSRP, trained observers visited the program and control group classrooms in the 
FOL-Newark site, but in FOL-Newark the full CLASS standardized assessment tool was used 
to assess not only the emotional climate but also the instructional climate of the classrooms and 
the amount of instructional time that teachers provided to children. (See Appendix C for a 
description of these measures.) Observers, who were blind to classroom random assignment 
status and were not part of the research team, rated classrooms on a scale of 1 to 7 on each of 
the CLASS dimensions, with 1 to 2 indicating low scores, 3 to 5 indicating moderate scores, 
and 6 to 7 indicating high scores. Observations in the program group classrooms occurred on a 
day that the CCC was not in attendance to keep observers blind to the treatment condition and 
to understand how the teachers conducted the class independently. 


5 Raver et al. (2009b). 

6 Notably, effects differed somewhat depending on the model specification, with more stringent models 
showing fewer impacts; see Raver et al. (2011). 

7 Zhai, Raver, and Jones (2012). In further analyses using propensity score matching, CSRP did show posi- 
tive kindergarten impacts for children who attended higher-quality schools compared with children who had 
similar propensity scores and attended lower-quality schools. 

8 Morris et al. (2010). 
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Box 4.1 


How to Read the Impact Tables in This Report 

In the context of this demonstration, an “impact” is a measure of the effect of the 
intervention, represented by the difference between those teachers and children in centers 
that were randomly assigned to the program group compared with those that were randomly 
assigned to the control group. All the tables in this chapter that show impacts use a similar 
format, with information provided for teachers, classrooms, and children in each of the 
program and control groups. Because centers were assigned randomly either to the 
program group or to the control group, the effects of the program can be estimated by 
looking at the difference in outcomes between the two groups, controlling for the grouping 
of children within classrooms and centers and teachers within centers. 

The “Difference” column in the tables shows the differences between the two groups’ 
outcomes — that is, the intervention’s estimated impacts on the outcomes. The asterisks 
show the probability that this difference, or impact, arose by chance, with one asterisk 
indicating a 10 percent probability, two asterisks indicating a 5 percent probability, and 
three asterisks indicating a 1 percent probability. 

The “effect size” creates a common metric to compare impacts for measures that are based 
on different scales. The effect size allows comparisons of impacts across studies and 
provides some indication of the magnitude of the impact. 

An examination of subgroup impacts requires an additional test of statistical significance to 
assess the magnitude of differences in impacts across subgroups. Therefore, an additional 
column, “Difference Between Subgroup Impacts,” is included in the tables showing sub- 
group impacts. Similar to the asterisks described above, the daggers in this column corre- 
spond to the level of statistical significance. One dagger corresponds to the 10 percent level; 
two daggers, the 5 percent level; and three daggers, the 1 percent level. Whenever such 
differences are statistically significant, one can have greater confidence that the underlying 
impacts for the subgroups involved are actually different. 


• Based on observers’ ratings in FOL-Newark, teachers in the program 
group had more positive interactions and fewer sarcastic and angry in- 
teractions and were more sensitive and better able to prevent misbehav- 
ior than teachers in control classrooms; instructional time also increased 
in the program group classrooms. 

As shown in Table 4.2, the FOL program group classrooms were rated significantly 
better than the control group classrooms on a composite measure of positive classroom man- 
agement — a measure that reflects teachers’ positive affect with children, lack of sarcasm and 
anger, and ability to comfort children and prevent misbehavior by setting clear expectations and 
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Box 4.2 


Understanding and Contextualizing Effect Sizes 

The effect size is used to compare the impacts in this study with those from other studies, 
yielding an assessment of the magnitude of the impact for different measures. Technically, the 
effect size indicates the proportion of a change in standard deviation that is caused by the 
intervention. 

How large are these effects? FOL’s effect sizes on children (of approximately one-third of a 
standard deviation) are in the range of those of other studies that have evaluated enhancements 
to an existing preschool curriculum. For example, in the Preschool PATHS evaluation, effect 
sizes on proximal child outcomes were approximately one-third of a standard deviation, with 
larger effect sizes for teacher-report outcomes;* in the Head Start RED1 trial, effect sizes on 
children were as large as one -third of a standard deviation, with effects on targeted outcomes 
typically in the 0.2 range;* in the Tools of the Mind evaluation, effect sizes on outcomes were 
as large as 0.5 for teacher-reported behavior, but ap-proximately one-fourth to one-third of a 
standard deviation for standardized test scores.* 

Comparisons with other widely cited preschool evaluations, such as Perry Preschool and the 
Head Start Impact Study, are more difficult. 5 These studies tested a particular fomi of pre- 
school, rather than an enhancement to an existing preschool curriculum, and therefore are 
comparing differences between groups of children with very different care experiences. 
However, as a point of reference, FOL’s impacts on children of a little more than one-third of a 
standard deviation are neither as small as those of Head Start (about 0.15 of a standard devia- 
tion) nor as large as those found in Perry, which is often touted as a re-sounding success (and 
which on some measures had effect sizes as large as 1 standard deviation). 

NOTES: *Domitrovich, Cortes, and Greenberg (2007). 

Bierman et al. (2008). 

'Barnett et al. (2008). 

§ Wickart, Bond, and McNeil (1978). 


using effective praise. The effect size of 0.75 indicates a relatively large improvement in 
teachers’ behavior on this measure — with scores improving from 5.2 (for the control group) to 
5.8 (for the program group), on average, on the seven-point scale. This finding implies that the 
first hurdle for the intervention was cleared: the FOL training improved teachers’ classroom 
management skills and the emotional climate of the classrooms. 

Teachers in FOL program group classrooms also scored higher on measures of their 
management of classroom time, with an effect size of 0.63. In addition, the amount of instruct- 
tional time was significantly longer in the FOL classrooms, by an average of 10 minutes out of a 
120-minute observation period. If such gains were representative of gains achieved every 
weekday, this would translate to 50 minutes more instruction per week, or a week’s more 
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The Foundations of Learning Demonstration 
Table 4.2 


Newark Program Impacts on Observed Ratings of Teacher Behavior in the 
Classroom, Pre-Kindergarten Year 


Outcome 

Mean Score" 1 
Program Control 
Group Group 

Difference 

(Impact) 

Standard 

Error 

Effect 

Size 

Positive classroom management 

Composite 

5.8 

5.2 

0.6 ** 

0.3 

0.75 

Positive climate 

5.6 

5.0 

0.6 

0.4 

0.60 

Negative climate b 

1.1 

1.8 

-0.6 *** 

0.2 

-0.90 

Teacher sensitivity 

5.2 

4.8 

0.4 

0.3 

0.46 

Behavior management 

5.4 

4.7 

0.8 ** 

0.4 

0.72 

Use of classroom time 

Management of classroom time 

5.4 

4.9 

0.5 * 

0.3 

0.63 

Amount of instructional time (minutes) 

35.6 

25.1 

10.6 ** 

4.4 

0.96 

Oualitv of language instruction 

Composite 

4.3 

3.8 

0.5 

0.3 

0.56 

Regard for student perspectives 

5.1 

4.9 

0.2 

0.3 

0.28 

Use of engaging teaching methods 

4.2 

3.5 

0.6 * 

0.3 

0.61 

Promoting understanding through conversation 

3.5 

3.0 

0.4 

0.4 

0.44 

Encouragement of students' language use 

4.3 

3.6 

0.7 

0.5 

0.54 

Sample size 

26 

25 





SOURCES: MDRC calculations using classroom observations in September-October 2007 and April-May 2008. 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The table presents adjusted means that control for random assignment blocks and baseline (fall) Classroom 
Assesssment Scoring System (CLASS) dimension scores. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the control 
group. 

a For each CLASS dimension observers rated classrooms on a scale from 1 to 7, with 1 representing "low" and 7 
representing "high." Amount of instructional time is measured in minutes rather than scored. 
b "Negative climate" is reverse-coded for the composite score. 


instruction over a school year . 9 These improvements are consistent with a central hypothesis of 
the demonstration — that children with challenging behaviors may divert teachers from 
providing instruction to all children and that addressing teachers’ classroom management skills 
would reduce unproductive time in preschool classrooms. 

The increased instructional time in FOL-Newark did not, however, lead to statistically 
significant improvements overall in the quality of language instruction that children received. At 
the outset of FOL, it was not clear whether the intervention would enable teachers to engage in 


9 This concluson is based on calculations for a 40-week year. 
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better instruction or whether the focus on behavior management would interfere with instruc- 
tion. Neither appeared to be true, as statistically significant differences in instructional quality 
were very rare. 

Descriptive Analysis of Teacher’s Behavior One Year Later 

As presented in a previous report of the FOL-Newark site, 10 there was also interest in 
whether the program group teachers sustained their changes in practice into the following 
school year, after the intensive training and coaching support they received during the year the 
program implementation had ended. To address this question, observations were conducted of 
FOL-assigned lead teachers who were still teaching preschool the following year, most of 
whom were provided with a very short refresher course. Unlike the analyses described elsewhere 
in this chapter, these analyses compare FOL teachers’ classroom practices in spring of the 
follow-up year with their own performance one year earlier, near the completion of their partici- 
pation in the intervention. Thus, these observational findings do not reflect impact analyses (or 
comparisons of the program and control groups), and they should not be interpreted as such. 

The results from this descriptive analysis are quite positive. In most cases, there were no 
differences in the classroom behavior observed during the intervention year and a year later, 
with teachers showing similar levels of positive classroom management, management of 
classroom time, and instruction. Indeed, even overall student engagement was unchanged from 
the prior year. In addition, teachers who were initially enrolled in the program group continued 
to show further improvements in their ability to maintain positive classroom climate, their 
sensitivity to children, and their use of engaging teaching methods relative to their own perfor- 
mance in the spring of the prior academic year, which was not expected. Only on one measure 
did teachers show reductions in the benefits of the program from the prior year, with small 
increases in teachers’ emotionally negative climate. These findings suggest that teachers in 
Newark, surprisingly, did appear to sustain practice in the year following the intervention, 
showing that investments in their professional development in one year may affect successive 
cohorts of the children they teach in preschool, a point that is discussed in Chapter 5 with regard 
to the costs and benefits of the program. 

Impacts on Child Behavior in the Classroom 

While FOL succeeded in changing the way teachers managed their classrooms, the next 
question that was addressed in FOL-Newark was whether exposure to the program would lead to 
improvements in children’s behavior in the classroom. To address this question, an observa- 


10 Morris et al. (2010). 
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tion team rated a subset of children on their conflicts and positive interactions with teachers and 
peers; 11 as described in Appendix C, the team also assessed the extent to which children were 
engaged in classroom activities, using the same l-to-7 scale that was used for the classroom 
observations, 12 and teachers were asked to rate children’s behavior. 

• For children in FOL-Newark, the intervention led to reductions in con- 
flicts with teachers and peers and higher levels of engagement in the 
classroom. 

Findings on children’s observed behavior are shown in Table 4.3. As shown in the first 
panel under “Problem behavior,” FOL led to statistically significant reductions in children’s 
conflicts with teachers and peers, but the intervention did not otherwise change the quality of 
teacher-child or peer interactions. In addition, children in the FOL-Newark program group 
demonstrated greater levels of engagement in the classroom and showed greater ability to 
control their behavior. Effect sizes on these impacts are moderate, ranging from 0.27 to 0.34 for 
child-specific outcomes and 0.60 for overall student engagement. Not surprisingly, these effect 
sizes are slightly smaller than those for the classroom impacts discussed above; this is consistent 
with the theory of change of the intervention, with the most proximal target of the intervention 
(teacher behavior) expected to demonstrate the largest impact. 

Notably, while observers in program classrooms found lower levels of conflict and 
greater engagement in classroom activities, the same was not true for teachers, who reported no 
differences in children’s behavior or skills between the program and control groups; impacts on 
these measures are shown in the bottom panel of Table 4.3, under “Teacher reports.” One 
hypothesis is that the training that teachers received primed them to see challenging behaviors, 
even as it increased their capacity to effectively manage those behaviors when they occurred. In 
addition, while not shown in the tables, FOL led to a statistically significant reduction in the 
prevalence of children’s challenging behaviors, as reported by teachers when they were asked 
about the frequency of each of a number of behaviors in their classrooms as a whole rather than 
about individual children. 


n Five children in each classroom were selected for the observations based on their gender and baseline 
behavior problems score as rated by teachers. More specifically, the subsample was selected to ensure an equal 
number of boys and girls overall, as well as representation of children at low, moderate, and high levels of 
behavior problems at preschool entry. 

12 Downer et al. (2010). 
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The Foundations of Learning Demonstration 
Table 4.3 


Newark Program Impacts on Observed and Teacher 
Ratings of Child Behavior, Pre-Kindergarten Year 


Outcome 

Mean Score 

Program Control Difference 
Group Group (Impact) 

Standard 

Error 

Effect 

Size 

Observations 3 






Problem behavior 






Teacher conflict 

1.2 

1.5 

q 2 *** 

0.1 

-0.40 

Peer conflict 

1.4 

1.6 

-0.2 * 

0.1 

-0.27 

Approach to learning 






Task engagement 

4.9 

4.6 

0.2 * 

0.1 

0.31 

Task self-reliance 

3.1 

3.1 

-0.1 

0.2 

-0.07 

Task behavior control 

5.4 

5.1 

0.3 * 

0.2 

0.34 

Overall classroom student engagement 

5.7 

5.2 

0.6 * 

0.3 

0.60 

Positive social behavior 






Teacher communication 

2.2 

2.4 

-0.2 

0.1 

-0.20 

Teacher positive engagement 

3.2 

3.4 

-0.2 

0.2 

-0.27 

Peer communication 

2.5 

2.6 

-0.1 

0.2 

-0.14 

Peer sociability 

3.4 

3.5 

-0.1 

0.1 

-0.11 

Peer assertiveness 

2.1 

2.3 

-0.2 

0.2 

-0.21 

Teacher reDorts b 






Problem behavior 






Internalizing problems 

2.7 

2.3 

0.4 

0.6 

0.11 

Externalizing problems 

4.1 

3.7 

0.4 

0.7 

0.08 

Teacher-student conflict 

12.4 

12.3 

0.1 

0.9 

0.02 

Attention problems 

3.6 

3.5 

0.1 

0.6 

0.02 

Approach to learning 






Work-related skills 

4.8 

4.8 

0.1 

0.1 

0.08 

Positive social behavior 






Compliance with teachers' directives 

4.0 

4.0 

0.1 

0.1 

0.08 

Social competence 

4.0 

4.0 

0.0 

0.1 

0.06 

Teacher-student closeness 

34.5 

35.8 

-1.3 

0.9 

-0.24 

Early academic skills 






Language and literacy skills 

3.9 

3.6 

0.3 

0.2 

0.27 

Math knowledge 

3.7 

3.6 

0.1 

0.2 

0.05 

General knowledge 

3.9 

3.7 

0.3 

0.2 

0.28 

Sample size 






Observations of students 

130 

121 




Teacher reports on students 

283 

248 




Classrooms 

26 

25 





(continued) 
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Table 4.3 (continued) 


SOURCE: Based on MDRC calculations of classroom observations and teacher surveys. 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Regression-adjusted means control for random assignment status and blocking, baseline Classroom Assessment 
Scoring System (CLASS) measures, and baseline child characteristics. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the control 
group. 

a The observed outcomes under "Problem behavior," "Positive social behavior," and "Approach to learning" are 
Individualized CLASS (InCLASS) observations. "Overall classroom student engagement" is a CLASS measure. 
For each dimension, observers rated children and classrooms on a scale from 1 to 7, with 1 representing "low" and 
7 representing "high." 

b Teacher-reported outcomes control for the child's baseline score on a given measure, when available. These 
include baseline measures for the Cooper-Farran Behavioral Rating Scales (CFBRS), the Behavior Problems Index 
(BPI), and the Positive Behavior Scale (PBS). "Internalizing problems" and "Externalizing problems" refer to the 
internalizing and externalizing subscales of the BPI. "Teacher-student conflict" and "Teacher-student closeness" 
refer to the Student-Teacher Relationship Scale conflict and closeness subscales. "Attention problems" refers to the 
Caregiver-Teacher Report Form attention problems subscale. "Compliance with teachers' directives" and "Social 
competence" refer to the PBS compliance and social competence subscales. "Work-related skills" refers to the 
CFBRS work-related skills subscale. "Early academic skills" includes the subscales of the Academic Rating Scale. 


Findings from Foundations of Learning in Chicago 

FOL-Chicago focused on outcome measures that were not obtained in Newark, by collecting 
information from assessments that were conducted directly with the children. These assess- 
ments measured children’s executive function (a combination of their attention, inhibitory 
control, and memory, underlying their approach to learning), children’s social problem-solving 
skills, and their early academic skills (math and pre-literacy). (See Appendix C for further 
explanation.) As discussed below, the impacts of FOL-Chicago varied across outcomes as- 
sessed, as well as by the source of that information. More specifically, there is some evidence of 
positive impacts on children’s approaches to learning (executive function), but not on outcomes 
that were less centrally targeted by the intervention (social problem-solving skills and early 
academic skills). 

• In FOL-Chicago, the intervention led to improvements in children’s ap- 
proaches to learning (children’s ability to engage in the learning tasks of 
preschool). 

As shown in Table 4.4, interviewers evaluated children on three tasks designed to as- 
sess their executive function — Head to Toes, in which children are asked to point to their 
heads or their toes when the interviewer does the opposite; Pencil Tap, in which children are 
asked to tap once with a pencil when the interviewer taps twice, and vice versa; and Gift Wrap, 
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The Foundations of Learning Demonstration 
Table 4.4 

Chicago Program Impacts on Approaches to Learning, Pre-Kindergarten Year 


Outcome 

Mean Score 

Program Control Difference 
Group Group (Impact) 

Standard 

Error 

Effect 

Size 

Executive Functioning (assessed) 






Head to Toes (gross motor) 3 

6.7 

4.4 

2.3 * 

1.24 

0.37 

Pencil Tap (fine motor) b 

0.6 

0.6 

0.0 

0.07 

0.05 

Gift Wrap (measured in seconds) 

42.5 

34.2 

8.3 * 

4.23 

0.34 

Positive engagement (observed) 3 

2.1 

1.8 

0.2 

0.13 

0.38 

Attentiveness/inhibitory control (observed) 11 

2.5 

2.4 

0.0 

0.10 

0.06 

Sample size 






Students (total = 307) 

155 

152 




Classrooms (total = 40) 

20 

20 





SOURCE: Based on MDRC calculations of direct child assessments in April-May 2008. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Regression-adjusted means control for random assignment status and matched pairs and select baseline 
child, teacher, and site characteristics. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the 
control group. 

a Scores on this measure may range from 0 to 20, with 0 representing no success and 20 representing 
success across the task's 10 trials. 

b Scores reflect the percentage of correct responses across the task's 16 trials. 

c This outcome represents an average score of 6 items on a scale of 1 to 4. 

d This oucome represents an average score of 16 items on a scale of 1 to 4. 


in which children are asked to “not peek” while a gift is noisily wrapped behind them. These 
tasks are critical because they reflect the same skills that children need to successfully engage in 
classroom activities — attention, inhibitory control, and working memory — and have been 
validated extensively in their association with concurrent classroom behavior and long-term 
academic achievement. 13 As shown in Table 4.4, in two of these three tasks (Gift Wrap and 
Head to Toes), FOL-Chicago had statistically significant impacts. The effect sizes of these 
impacts were 0.34 and 0.37, respectively, showing moderate differences between program and 


13 Blair, Granger, Razza (2005); Blair and Diamond (2008). 
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control groups on these assessments (and impacts that were similar in size to those reported by 
trained observers of children’s behavior in FOL-Newark). Interviewers also rated children’s 
“positive engagement” and “attentiveness/inhibitory control” based on the children’s behavior 
during these activities. Statistically significant program impacts were not found for either of 
these measures. In addition to children’s executive function skills, interviewers in FOL-Chicago 
also assessed children on their social problem-solving skills (children’s ability to generate 
solutions in response to negative peer interactions that reduce rather than exacerbate the 
conflict) and their early academic skills (language, literacy, and math), none of which was 
directly targeted by the program, as shown in Table 4.5 and explained in Appendix C. To assess 
their social problem-solving skills, teachers showed the children pictures and described vi- 
gnettes of peer conflicts. The children were rated on the extent to which their responses were 
competent (prosocial or seeking adult support as compared with aggressive or avoidant). There 
were no differences in children’s responses to stories in FOL and control classrooms. On a 
receptive language test (assessing children’s understanding of words using the Peabody Picture 
Vocabulary Test, or PPVT) 14 and on a literacy test (assessing children’s knowledge of letters, 
sound-letter correspondence, and early reading skills using the Woodcock Johnson Letter-Word 
Identification subtest), 15 there were no differences between children in FOL and control class- 
rooms. On the math skills portion of the same test (assessing children’s early understanding of 
counting, size, money, time, addition, and subtraction on the Woodcock Johnson Applied 
Problems subtest), children in FOL classrooms outperformed their peers in control classrooms, 
although the difference fell short of statistical significance (with an effect size of 0.49). 

As in FOL-Newark, teachers in FOL-Chicago were asked to assess children’s behavior 
in the classroom on measures directly targeted by the intervention (problem behavior and 
approach to learning) as well as on indirect targets (academic skills and social problem-solving 
skills). Similar to FOL-Newark, while some benefits were found based on information that was 
collected by trained assessors, there were no impacts in the teacher-reported data. (These 
findings are shown in Appendix Table C.3.) 


Subgroups of Children 

In both Newark and Chicago, analyses were conducted to detect whether the impacts differed 
for several key subgroups: (1) for children at different levels of behavioral risk; (2) by children’s 
age (for three-year-olds compared with four-year-olds in Chicago); (3) by gender; and (4) by 
children’s race/ethnicity (for Hispanic and black, non-Hispanic in Newark). These analyses are 


14 Dunn and Dunn (1997). 

15 McGrew, Schrank, and Woodcock (2007). 
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The Foundations of Learning Demonstration 
Table 4.5 


Chicago Program Impacts on Secondary Child Outcomes 
Assessed, Pre-Kindergarten Year 



Mean Score 





Program 

Control 

Difference 

Standard 

Effect 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Social problem-solving skills 

Adaptive problem solving 11 

8.9 

8.8 

0.1 

0.38 

0.06 

Early academic skills 15 

Peabody Picture Vocabulary Test 
Woodcock Johnson 

85.4 

81.7 

3.7 

5.40 

0.24 

Early math knowledge 

99.4 

93.9 

5.4 

2.98 

0.49 

Early language/literacy skills 

101.5 

98.8 

2.7 

2.88 

0.20 

Sample size 






Students (total = 306) 

154 

152 




Classrooms (total = 40) 

20 

20 





SOURCE: Based on MDRC calculations of child assessments in April-May 2008. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Regression-adjusted means control for random assignment status and matched pairs and select 
baseline child, teacher, and site characteristics. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the 
control group. 

a The problem-solving subscale of the Challenging Situations Task rates children's responses to 4 
hypothetical situations. Scores range from 4 (more negative) to 12 (more positive). 

b All test scores are standardized, based on child's age and raw score, with a mean of 100 and a 
standard deviation of 15. 


presented in Appendix C. Overall, there were only limited statistically significant differences in 
impacts across the subgroups that were examined. 


Putting the Impact Findings Together Across All Three Studies 

One of the strengths of this body of research is that the same program model was evaluated in 
three different random assignment evaluations: two cohorts in Chicago (CSRP and FOL- 
Chicago) and one in Newark (FOL-Newark). Such rich information provides a more complete 
picture of impacts than is typical in intervention studies. Similar findings across the sites on 
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related constructs provide greater evidence of program impacts, if they exist. While precisely 
the same data were not collected in each trial, comparing and contrasting the results provides 
critical information to begin to build an understanding of the pathways by which the FOL 
model played out in temis of impacts on classrooms and children. 

This cross-site comparison was conducted in two ways: First, impacts were compared 
and contrasted across the sites, “eyeballing” the pattern of impacts (shown in Figure 4.2). Having 
two or more studies with statistically significant impacts for a single construct provides greater 
confidence in the findings than if the findings are found in only a single site. In Figure 4.2, 
favorable statistically significant effects are denoted by plus signs (+) and no effect as zero (0); 
constructs that were not measured are indicated by “NA” (not available). Second, where possi- 
ble, the data across sites were pooled and impacts were estimated on this combined sample. 
Given concerns about statistical power in CSRP and FOL-Chicago because of small sample 
sizes, the pooled results provide some confirmation of the results of the individual studies. 

• Across studies, the intervention changed the way teachers interact with 
students in their classrooms. 

In tenns of classroom outcomes, impacts in CSRP and FOL-Newark show that the pro- 
gram, as intended, changed the way teachers interact with their students in their classrooms. In 
both sites, the intervention was found to improve teachers ’positive classroom management, the 
key domain on which teachers were trained. The similarity in results across the two sites where 
these data were collected and the fact that implementation findings show participation in 
training at high levels in all three sites bolster confidence in these results. These findings show 
that this training and coaching model is effective in changing its primary target of teachers’ 
ability to sensitively manage problem behavior in their preschool classrooms. The effect sizes in 
this domain are sizeable — ranging from approximately 0.5 to nearly 1.0. 

While findings from FOL-Newark suggest that this improvement in positive classroom 
management led to increased instructional time, there is no corroborating evidence in the other 
two sites on this point, as neither CSRP nor FOL-Chicago collected this infonnation. The 
finding is consistent with the hypothesis behind the intervention — that children with challeng- 
ing behaviors might distract teachers from being able to find time for instruction — but this 
finding must be replicated to establish confidence in its role as a potential link in a theory of 
change that relies on improved classroom management to improve academic outcomes for 
children. That said, the finding is important in suggesting that FOL may increase the opportuni- 
ty for children’s learning. 
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The Foundations of Learning Demonstration 
Figure 4.2 


Summary of Pre-Kindergarten Findings Across Studies: CSRP, FOL-Newark (FOL-N), FOL-Chicago (FOL-C) 


Teacher and Classroom Outcomes 

Child Outcomes 

Directly Targeted 

Secondary 

Directly Targeted 

Secondary 

Positive Classroom Management 

Classroom Productivity and 

Problem Behavior 

Positive Social Behavior 

CSRP: + 

Amount of Instructional Time 

Conflict/Extemalizing Behavior 

(observed) 

FOL-N: + 

CSRP: NA 

Problems (observed) 

CSRP: NA 

FOL-C: NA 

FOL-N: + 

CSRP: + 

FOL-N: 0 


FOL-C: NA 

FOL-N: + 

FOL-C: NA 


Quality of Instruction 

FOL-C: NA 

Early Academic Skills 


CSRP: NA 

Conflict/Extemalizing Behavior 

CSRP: + 


FOL-N: 0 

Problems (teacher report) 

FOL-N: NA 


FOL-C: NA 

CSRP: + 

FOL-C: 0 



FOL-N: 0 

FOL-C: 0 

Social Problem-Solving Skills 

CSRP: NA 



Approaches to Learning 

FOL-N: NA 



Executive Functioning 

FOL-C: 0 



CSRP: + 

FOL-N: NA 
FOL-C: + 




Attentiveness/Behavior Control 




CSRP: + 

FOL-N: + 

FOL-C: 0 




Positive Engagement 




CSRP: " 0 
FOL-N: + 

FOL-C: 0 



NOTE: A plus sign (+) indicates that there was a statistically significant impact on that outcome; a zero (0) indicates that there was no statistically significant 
impact; and “NA” indicates that the outcome was not measured in the given study. 







• Across the three studies, the intervention consistently improved chil- 
dren’s approaches to learning. There is also promising evidence that the 
intervention reduced children’s problem behavior. 

In tenns of outcomes for children, both CSRP and FOL-Newark show benefits in re- 
ducing children ’s problem behavior, with each reducing observations of children’s conflicts 
with peers. Moreover, the intervention had the most consistent impacts on children ’s approach- 
es to learning, with evidence of positive impacts observed across all three studies. In Newark, 
positive impacts are found on ratings based on observations of children in the classroom; in 
CSRP and in FOL-Chicago, positive impacts are found on children’s executive function skills 
based on direct assessments. The data that were collected in CSRP and FOL-Chicago are 
especially powerful, as the executive function outcomes assessed in these sites have been shown 
in other research to be associated with children’s ability to engage in the learning tasks of school 
as well as with children’s academic skills. 16 Pooled results of CSRP and FOL-Chicago (not 
shown in the tables) confinn these cross-study impacts, showing statistically significant im- 
provements in children’s executive function tasks. 

• There was no clear evidence that the intervention had a positive effect 
on secondary child outcomes, such as early academic skills and positive 
social behavior, which were less directly targeted. 

It was originally hoped that impacts on children’s problem behavior and approaches to 
learning might influence other outcome domains such as positive interactions with peers and 
especially early academic skills. While there was some initial positive evidence of this in CSRP, 
significant impacts on these secondary outcomes were not found in either of the FOL sites. 
Pooled results of CSRP and FOL-Chicago show no statistically significant impacts on measures 
of children’s early academic skills. As discussed in more detail in Chapter 6, perhaps the focus 
of this intervention is not sufficient to boost such skills. 

On the one hand, these results are encouraging: they suggest that the program changed 
teacher behavior as intended, and this led to some improvements in children’s approaches to 
learning and problem behavior — the outcomes for children that were most centrally targeted 
by the intervention. At the same time, it was hoped that the greater time for instruction would 
increase learning, and the positive effects on executive function skills and related approaches to 
learning would increase early academic skills, but these impacts were not observed. 


16 Blair, Granger, Razza (2005); Blair and Diamond (2008). 
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Looking Ahead: A Glimpse at the Children as They Make the 
Transition into Elementary School 

All three studies collected limited information about how children fared when they entered 
elementary school. In CSRP and FOL-Newark, the infomiation was gathered through surveys 
that were conducted with kindergarten teachers. In FOL-Chicago, the information was collected 
through school records. Since the kindergarten and first grade data reflect not only children’s 
behavior and skills, but also teachers’ perceptions and school policies, they are much less 
reliable than the preschool data collected by trained observers and interviewers. 

Notably, children were widely dispersed in the year following the intervention, from 20 
to 50 preschools in each of the demonstration studies to 100 to 150 schools the following year. 
As a result, the children from the intervention classrooms entered elementary school classrooms 
with many children who had not experienced the intervention model. 

Based on teacher reports, FOL-Newark had very few sustained impacts on children 
the year after they received the intervention. Surprisingly, the only statistically significant 
effects were observed in the negative direction — with higher levels of internalizing and 
externalizing problems on the part of children who came from program group preschools 
compared with those who came from control group preschools, as shown in Appendix Table 
C.9. This may have been a chance finding or may reflect children’s difficulty adjusting to less 
supportive environments. Similarly, despite these benefits during the preschool intervention 
year, there were no sustained effects in CSRP, on average, for children’s skills when they 
entered kindergarten. 

In FOL-Chicago, similar findings emerged, with no sustained effects of FOL, although 
the information that was provided by school records is quite sparse. Results for the four-year- 
old children (who were old enough to enter kindergarten in the year following FOL) are 
presented in Table 4.6. In kindergarten, children were presented with an early literacy test that 
was scored by teachers (called the Dynamic Indicators of Basic Early Literacy Skills, or 
DIBELS), which provided both a mean score and a recommendation for instructional support. 
A sizeable 10 to 15 percent of the children were rated as being in need of intensive instructional 
support on this measure. School records also provided information on regular attendance and 
whether children had a disability based on the presence of an Individualized Education Plan 
(IEP). Rates on this latter measure were especially low — only 5 to 8 percent of children had a 
learning disability or social-emotional disorder and only 1 to 2 percent reportedly had a social or 
emotional disorder. FOL had no impacts on measures of literacy skills, attendance, or disability 
status for these children. 

Emerging research in the field suggests that the quality of elementary schools may play 
a significant role in the longer-term impacts of an intervention. Given the dispersion of children 
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The Foundations of Learning Demonstration 
Table 4.6 


Chicago Program Impacts on Child Outcomes, 
Kindergarten and First Grade 


Outcomes 

Mean Score 

Program Control Difference 
Group Group (Impact) 

Standard 

Error 

Effect 

Size 

First vear of school follow-UD (kindemarten) 






Early literacy standardized score 3 

-0.1 

0.1 

-0.1 

0.14 

-0.20 

Recommended for intensive instructional support (%) 

14.6 

9.6 

4.9 

6.15 

0.16 

Poor attendance 6 (%) 

22.3 

15.2 

7.1 

5.31 

0.19 

Disability (%) 






Social-emotional disorder 

0.9 

1.7 

-0.9 

1.49 

-0.07 

Social-emotional or learning disability 

7.6 

5.4 

2.2 

5.97 

0.10 

Second vear of school follow-UD (first sirade) 






In kindergarten 6 (%) 

-2.0 

5.9 

-7.8 

4.87 

-0.48 

Average of all academic grades 6 

3.8 

3.8 

0.0 

0.16 

0.04 

Average of reading, math, and writing grades 6 

3.6 

3.5 

0.1 

0.18 

0.05 

Poor attendance 6 (%) 

16.9 

19.0 

-2.1 

5.23 

-0.05 

Disability (%) 






Social-emotional disorder 

0.7 

2.5 

-1.8 

1.78 

-0.12 

Social-emotional or learning disability 

7.8 

8.2 

-0.4 

5.72 

-0.01 

Sample size (total = 267) 

133 

134 





SOURCE: Based on MDRC calculations from Chicago Public School student records. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Regression-adjusted means control for random assignment status, matched pairs, and baseline child 
characteristics. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the control 
group. 

a Scores from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) are presented as standardized 
scores with a mean of 0 and a standard deviation of 1 . 

b "Poor attendance" means that the child was absent 10 percent of the time or more during the school year 
(measured in days). 

c "In kindergarten" refers to those who entered kindergarten a year older than their typical peers based on 
Chicago Public Schools' age requirements (n = 2) and children who were retained in kindergarten (n = 1). 

d All academic grades average includes Chicago Reading Framework, Mathematics Standards, Writing 
Standards, Listening Standards, Speaking Standards, Science Standards, and Social Science Standards. Grades 
are based on a scale of 1 to 5, where 1 = F and 5 = A. 

e Grades are based on a scale of 1 to 5, where 1 = F and 5 = A. 
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in Chicago, publicly available school-level data were used to investigate this further. Appendix 
Table C.10 uses achievement test scores (specifically, the percentage of students receiving a 
passing score in math and reading tests) as an indicator of school quality. However, among the 
limited outcome data available, there was no pattern of differences in impacts among students 
based on the quality of the elementary school they attended for kindergarten. 

In first grade, children’s grades in the academic subjects were available, with grades re- 
flecting children’s achievement in reading, math, writing, listening, speaking, science, and 
social studies. An average grade of 3.8 was just below a “B,” based on a scale of 1 to 5 (where 1 
= F and 5 = A). The second panel of Table 4.6 illustrates that, as in the first-year follow-up, 
FOL had no impacts on achievement or on attendance rates in this second post-intervention 
year. Notably, only three children delayed the start of kindergarten by staying in preschool an 
additional year, and only one child was retained in kindergarten. The same is true for the 
presence of an emotional and behavioral disorder, where only two children in the sample are 
scored as having such a disability. On both of these measures, the very low rates make it 
difficult to detect program impacts (although impacts on both measures are in the direction of 
favorable effects of the intervention), suggesting it is simply too early to fully assess the long- 
tenn effects of this program. However, children typically do not complete the process for 
placement into separate programs like special education until second or third grade. 17 


In sum, findings from both Newark and Chicago show no evidence of sustained effects of the 
program in the early elementary school years. Notably, as discussed earlier, the information that 
was gathered for those years in both studies was substantially weaker than that collected during 
the preschool year. However, given the context of the typical urban elementary school, with a 
larger number of children, higher student-to-teacher ratios, and greater expectations of inde- 
pendence and discipline, the absence of positive effects is perhaps not surprising. 


17 National Association of Special Education Teachers (n.d.); The IRIS Center (n.d.). 
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Chapter 5 

Cost and Benefits of Foundations of Learning 


This chapter documents the full economic cost of the Foundations of Learning (FOL) interven- 
tion as implemented in Newark and Chicago. The chapter begins by analyzing the cost of 
delivering the FOL program and describing the implications for replicating it in other sites. The 
next sections outline the potential for FOL or similar interventions to generate dollar benefits 
that can be weighed against the program costs, taking into account challenges in assigning 
dollar values to the impacts that are presented in Chapter 4. 


Summary of Findings 

• At approximately $1,750 per child, FOL represented a 14 percent in- 
crease in program costs over normal operating costs in Newark and a 
21 percent increase in Chicago. Although the average cost per classroom 
was higher in Chicago ($34,884) than in Newark ($26,873), the larger class 
size in Chicago meant that the per-child costs were similar in the two sites: 
$1,792 in Newark and $1,744 in Chicago. These costs are somewhat higher 
than the cost for the more typical approach of assigning a mental health 
consultant to intervene on an as-needed basis with children who exhibit dif- 
ficult behaviors. 

• Based on the limited data that are available on outcomes when the chil- 
dren from FOL classrooms went to elementary school, there is no evi- 
dence that the program’s monetary benefits will outweigh its costs. In 

nonexperimental research, measures of problem behavior and approaches to 
learning have been shown to be associated with improvements in these and 
other outcomes in the later school years. However, the limited follow-up data 
that were collected in the FOL studies did not show that benefits were sus- 
tained into the early elementary grades. In the case of some outcomes — for 
example, a child’s placement into special education — it is too soon to know 
whether the intervention had any effect, although, in this example, reductions 
in special education placements for even a small number of children could 
result in major savings that would outweigh the intervention’s cost. 
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Cost of the FOL Intervention 

The net cost analysis that is presented below captures the incremental costs of the new services 
that were provided in the FOL intervention in Newark and Chicago, over and above existing 
services that were available to the control sites. Using cost information that was collected by 
MDRC, the analysis disaggregates the cost of each key component of the intervention. All costs 
are expressed in 2010 dollars. Additional detail on the derivation of the cost estimates is 
provided in Appendix D. 

The financial costs of the FOL intervention services are disaggregated into three main 
components: 

• Teacher training: This includes the cost of compensating the trainers and 
any associated indirect or overhead costs, the cost of compensating the 
teachers and the Clinical Classroom Consultants (CCCs) for their time in the 
training, and any costs of the training materials. 

• Clinical classroom consultation: This includes the cost of CCCs’ orienta- 
tion, time spent modeling in the classroom and coaching teachers to reinforce 
the training, time spent providing targeted services to the highest-risk chil- 
dren and making referrals for them to outside services, and time spent in 
weekly clinical supervision. 

• Support for the intervention: This includes any ongoing technical assis- 
tance, advice, and support given to the trainers and CCCs to strengthen the 
intervention. It excludes the costs of activities that are conducted purely for 
data collection or other research purposes, which do not contribute to the 
quality of the intervention itself. 

For each of these components, the total cost of the FOL intervention is equivalent to the 
net cost because there is no analogous activity for classrooms in the control group. For example, 
as long as the teacher training does not substitute for additional training that teachers in the 
control group would have received, the total cost of the teacher training is all treated as a net 
new cost. However, if the control group received the services but received them at rates that 
differed from the FOL program group rates (for example, if children in the control group were 
referred to mental health services, but at lower rates than children in the program group), the 
costs for both the program and control groups would have to be estimated, with the net cost 
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being the difference between these two costs. However, no such differences in services used 
were found, based on data collected from teachers in both sites. 1 

Cost Analysis 

Table 5.1 reports the results of the cost analysis, shown separately for Newark and Chi- 
cago. The total cost for FOL teacher training, classroom consultation, and support or technical 
assistance is estimated to have been $698,699 for Newark and $697,675 for Chicago. The 
largest costs by far were for clinical consultation, which accounted for nearly three-fourths of 
the total costs of the intervention. Of the remaining costs, teacher training cost twice as much as 
the additional support for the intervention. 

Costs in FOL-Newark and FOL-Chicago were quite similar. In Chicago, costs were 
slightly higher for clinical consultation because CCC salaries were slightly higher there 
($51,700 compared with $44,000 in Newark); in Newark, costs were slightly higher for the 
teacher training because more staff participated in the training there (253 staff payments 
compared with 218 in Chicago). Yet these small differences offset each other and yield similar 
total estimates, such that the total costs came within $1 ,000 of each other across the two sites. 

Since the Newark program served 26 classrooms, whereas the Chicago program served 
20 classrooms, the average cost per classroom was lower in Newark — $26,873 compared with 


The Foundations of Learning Demonstration 
Table 5.1 

Estimated Program Costs, FOL-Newark and FOL-Chicago 


Component 

Total Cost 

Cost per Classroom 

Cost per Student 

Newark 

Chicago 

Newark 

Chicago 

Newark 

Chicago 

Teacher training ($) 

132,560 

109,956 

5,098 

5,498 

340 

275 

Clinical classroom consultations ($) 

507,638 

534,429 

19,525 

26,721 

1,302 

1,336 

Support for the intervention ($) 

58,501 

53,290 

2,250 

2,665 

150 

133 

Total for all components 8 ($) 

698,699 

697,675 

26,873 

34,884 

1,792 

1,744 


SOURCES: MDRC calculations based on fiscal and budget data. 

NOTES: Costs shown are for 26 classrooms in Newark with an average of 15 students per classroom, and 20 
classrooms in Chicago with an average of 20 students per classroom. All costs are shown in 2010 dollars. 

“Includes all teacher training expenditures, classroom consultant expenditures, and technical assistance 
provided by MDRC. 


'See Morris et al. (2010). 


83 







$34,884 in Chicago. Although the intervention essentially operated at the classroom level, it is 
also informative to examine the cost of the program per child served in the two settings. As 
shown in Table 5.1, average class size was 15 students in Newark and 20 students in Chicago. 
Thus, even though the per-classroom costs were higher in Chicago, the larger class size there 
relative to Newark means that the per-child costs were similar in the two sites: $1,792 in 
Newark and $1,744 in Chicago. Using estimates of program costs per child for each city, 2 these 
per-child costs represent a 14 percent increase in program costs in the Newark sites and a 21 
percent increase in program costs in the Chicago sites. 

Considerations for Replication 

The costs of FOL that are reported here represent the costs of adding FOL fully on top 
of existing training and professional development in preschool settings. Moreover, these 
estimates represent the full economic costs of program implementation, including the oppor- 
tunity cost of teacher training time and other resources (for example, meeting facilities). 
However, if FOL were integrated into an existing preschool program, the incremental financial 
cost of implementing the intervention could be reduced in a number of ways. 

For example, FOL training could be offered as part of existing professional develop- 
ment days and delivered on-site, thereby saving the costs that are associated with additional 
teacher training time and training space. Also, management costs might be reduced. But the 
largest costs of FOL are the salaries of the credentialed and skilled CCCs, for whom cost-saving 
options are not as clear. Notably, however, Head Start does have mental health consultants who 
serve individual children; these same staff members could be used in a coaching role in FOL, 
although that would substitute for their other activities and may result in a net reduction of the 
associated benefit. Given the findings on teachers’ sustaining practice (see Chapter 4), the 
estimated costs of implementing FOL that are presented in Table 5.1 represent a single-year 
investment in teachers that may have benefits that continue beyond one year. For example, 
subsequent cohorts of children, at least for the majority of preschool teachers who remain in the 
preschool setting from one year to the next, may also benefit from this program. 3 

The Challenge of Valuing Outcomes of the FOL Intervention 

Through improvements in educational outcomes, FOL has the potential to generate economic 
benefits for children who participate in the intervention, as well as for society as a whole, 
although positive impacts on outcomes that are easily monetized as benefits were not observed 

2 Bamett et al. (2010). Spending per preschool child in Newark under the Abbott regulations was $12,859; 
for Head Start in Chicago, spending per child was $8,413. 

’National preschool teacher turnover rates ranged from 25 to 50 percent in 2003 (Barnett, 2003). 
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as part of this evaluation. That is, positive impacts were not observed on measures of children’s 
pre-academic skills that have been linked repeatedly with longer-term outcomes in prior 
research; instead, positive impacts were observed on measures of executive function and 
classroom engagement, which have been shown to be associated with later benefits but that 
cannot be easily monetized in a cost-benefit study. 

In a benefit-cost analysis of a particular program or policy, social benefits and costs are 
typically disaggregated into three stakeholder groups. First, the government or public sector is 
one stakeholder and consists of all members of society in their role as taxpayers or citizens. 
Public sector benefits or costs may accrue at the federal, state, or local levels. Second, program 
participants may also experience private gains or losses as a result of the intervention. Third, in 
addition to any gains or losses to taxpayers as a group, other members of society (who have not 
participated in the intervention) may also experience economic benefits or costs as private 
individuals. Summing across these three stakeholder groups gives the perspective of society as a 
whole, including both public and private sector benefits and costs. 

Table 5.2 illustrates some of the educational outcomes that could be affected by FOL 
and the associated dollar benefits or costs that may accrue to different stakeholders, as well as 
the net economic effect for society as a whole (though, again, such outcomes were not actually 
observed in this demonstration). For example, if FOL were to reduce grade repetition, children 
would spend fewer years in the public education system, thereby saving state and local educa- 
tion expenditures. This cost savings represents a monetary benefit to the government sector and 
to society as a whole. Likewise, if FOL were to reduce the number of children in special 
education classes, the differential cost of special education over regular education classes would 
be a savings to government and a benefit to society as a whole. Or, if FOL were to increase 
teacher retention (not shown in table), the savings from fewer new teacher trainings and the 
increased value of more experienced teachers would be a savings to government and a benefit to 
society as a whole. Finally, if as a result of the FOL intervention more students eventually 
graduate from high school, there are a number of different potential economic consequences. If 
students spend more time in school instead of dropping out, education costs increase, which is a 
cost to the public sector and to society as a whole. However, as a result of their higher educa- 
tional attainment, FOL participants would be expected to have greater lifetime earnings, which 
benefits them as individuals and society as a whole. The higher earnings would also be associat- 
ed with increased tax revenue (for example, payroll and income taxes), a benefit to the public 
sector (the government) that is offset by the lost income to the individual in the taxes paid, with 
a net effect on society as a whole of zero. 
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The Foundations of Learning Demonstration 


Table 5.2 

Potential Monetary Benefits and Costs of Improved Educational Outcomes from FOL, by Stakeholder 





Stakeholder 


Outcome Affected 

Monetary Benefit or Cost 

Government 

Program 

Participants 

Rest of Society 
(did not 
participate in 
program) 

Society as a Whole 

Reduced grade repetition 

Fewer excess years spent in K-12 education 

+ 



+ 

Reduced use of special 
education 

Lower costs for special education 

+ 



+ 

Increased high school 
graduation rate 

(More years spent in K-12 education when 
dropping out is avoided) 

- 



- 


Increased lifetime earnings for participants 


+ 


+ 


Increased tax revenue to government from 
higher earnings 

+ 

- 


0 


SOURCE: Based on Karoly, Kilburn, and Cannon (2005), Table 4.1. 

NOTE: Parentheses denote costs as opposed to benefits. A plus sign indicates a monetary benefit; a minus sign indicates a cost; and a 0 indicates 
that the benefit(s) and cost(s) cancel each other out. 



Using this same framework, benefit-cost analyses of a number of early childhood inter- 
ventions have demonstrated the potential for high-quality early childhood programs to generate 
benefits to society that exceed the program costs. 4 Cost savings to government accrue from 
short-term effects related to the decreased need for special educational services and grade 
retention, as shown in Table 5.2. Other potential longer-term economic benefits to the public 
sector come from reductions in the use of social services and in criminality, and increased taxes 
paid on the gains in earnings. 5 That is, program participants benefit from higher income (net of 
taxes and transfers) and other members of society gain through the reduction in the social costs 
of crime. Summed across all stakeholders, early childhood programs can generate social 
benefits net of program costs that exceed zero. 

Much of the evidence of favorable economic returns to early childhood programs 
comes from interventions with long-term follow-up, with data on program participants’ experi- 
ences in adolescence, young adulthood, and even adulthood. 6 Participants in programs such as 
Perry Preschool, Abecedarian, and the Chicago Child-Parent Centers have been followed into 
their late twenties and beyond. At these ages, potential impacts can be observed for such 
outcomes as school performance (for example, grade repetition, special education use, high 
school graduation rates), antisocial behavior (for example, substance abuse, delinquency, 
crime), and economic outcomes (for example, employment and earnings, use of social services) 
— all of which can be more readily valued in dollar terms than the outcomes that were observed 
as part of the FOL demonstration. 

Benefit-cost analysis is also possible for early childhood programs without longer-term 
follow-up, but it is often more challenging to place an economic value on the types of outcomes 
that are typically observed when the intervention ends. Impacts on measures that are assessed in 
preschool or at kindergarten entry that are related to children’s cognitive development, social- 
emotional behavior, and other developmental measures are not as readily valued in economic 
terms as other outcomes, such as increases in earnings, reduction in crime, or the use of public 
services. However, drawing on evidence of the relationship between developmental measures in 
early childhood and outcomes later in life, 7 some studies have imputed the potential longer-term 
economic impacts that are associated with outcomes in early childhood. For example, evidence 
of impacts on achievement tests that is collected in the early elementary grades can be linked to 
subsequent educational outcomes such as high school graduation, which in turn affects labor 
market earnings. 8 However, such findings are strongest when they are based on data that are 


4 Karoly (2012). 

5 Belfield, Nores, Barnett, and Schweinhart (2006); Heckman et al. (2010); Reynolds et al. (2011). 

6 Karoly (2012). 

7 See, for example, Duncan et al. (2007). 

8 Aos et al. (2004); Aos et al. (2012). 
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collected in the early elementary grades for outcomes that have been studied extensively with 
regard to their long-term effects. 

Based on the impacts that are presented in Chapter 4, the FOL intervention led to posi- 
tive impacts on some of the outcomes measured but the monetary value of these impacts cannot 
be reliably translated into economic benefits. Significant impacts in the preschool year were 
found on measures of approaches to learning in CSRP and in the Newark and Chicago FOL 
sites, 9 and on problem behavior in CSRP and in FOL-Newark. (This was not assessed directly 
in Chicago.) In other work, such measures of approaches to learning and problem behavior have 
been shown to lead to long-term benefits for children. 10 However, the subsequent follow-up data 
that were obtained from available kindergarten and first-grade records in Chicago did not show 
any significant impacts on measures of school performance, such as test scores, grade repetition, 
or rates of special education use. Although these measures have been valued in other benefit- 
cost analyses of early childhood programs, the ability to identify impacts on grade repetition and 
special education was severely limited in the FOL study by the low frequency with which such 
events occurred by first grade in the Chicago Public Schools. Data that were reported by 
teachers in Newark also show few, if any, sustained benefits of FOL. It is not known whether 
the initial effects on approaches to learning or problem behavior were sustained into the early 
elementary grades in either site, since data were not collected on these measures specifically. In 
the absence of such follow-up data on these domains, dollar benefits were not assigned to the 
impacts on these measures as observed in the preschool year. 


The Potential for Economic Benefits from the FOL Intervention 

The findings to date for the FOL intervention in Newark and Chicago that are detailed in 
Chapter 4 do not provide a sufficient basis to calculate the dollar value of the program benefits 
that can be weighed against the program costs presented in Table 5.1. However, given the 
estimates of program costs, it is possible to consider how large the impacts on specific outcomes 
would have to be in either the short term or the longer tenn in order to cover the cost of the 
intervention and generate positive economic returns. 

For example, in one study the social cost of dropping out of high school measured over 
the individual’s lifetime but valued as of birth (the earliest point at which an intervention could 
begin) in terms of lost productivity, fringe benefits, and other nonwage public and private 
benefits (such as better health status and improved child rearing) was estimated to be $300,000 
to $450,000 in 2007 dollars. 11 This dollar figure is the present discounted value of the stream of 

9 CSRP was formerly the Chicago School Readiness Project. 

10 Zhai, Raver, and Jones (2012). 

1 'Cohen and Piquero (2009). 



future benefits. That is to say that the future dollars are discounted at each age to the value they 
would have as of birth to account for the fact that a dollar in the future is worth less today. Thus, 
out of 100 children served, if one fewer child drops out of high school as a result of the FOL 
intervention (reflecting a 1 percentage point increase in the high school graduation rate), the 
benefits to society that are spread across all 100 children who received the intervention would 
be $3,550 to $5,330 dollars per child participant, or about two to three times the per-child cost 
of the intervention shown in Table 5. 1 . 12 In other words, for each child who participates in FOL, 
the economic benefit to society would surpass the cost of the program. 

Is it likely that FOL would eventually produce an increase in high school graduation 
rates of 1 percentage point or more? By way of comparison, the Perry Preschool intervention, as 
of the age 40 follow-up, increased the high school graduation rate by 17 percentage points, 13 
while the Chicago Child-Parent Centers increased the rate by 7 percentage points as of age 26. 14 
Although it is unlikely that FOL would generate impacts of a similar magnitude as these two 
more intensive interventions (which measured the effects of a high-quality preschool program 
that was implemented at a time when the control group had little access to alternative early 
learning programs), it is possible that the favorable effects of FOL on approaches to learning 
could generate modest improvements in educational outcomes in later school years. 

In addition to improved high school graduation rates, early childhood interventions 
have demonstrated other educational benefits, such as reductions in special education use and 
reduced rates of grade retention, both of which are associated with substantial costs to the public 
education system. 15 A recent benefit-cost analysis of the Chicago Child-Parent Centers program 
provides estimates of these potential educational savings. 16 For example, a year of special 
education in the Chicago public school system, valued in 2007 dollars, is estimated to cost the 
public sector $9,910 in added educational costs, or $10,422 in 2010 dollars. Assuming that the 
savings from a reduction in special education use accrue at age 12 (the assumption made in the 
benefit-cost analysis of the Child-Parent Centers program), these savings at age 12 need to be 
discounted to age 4, the age at which the FOL program costs are incurred. The resulting present 
value savings is about $8,200 per year of special education costs averted (measured in 2010 
dollars). With a per-child cost of $1,744 (shown in Table 5.1), if FOL reduced special education 


12 This result can also be derived by noting that in Chicago, for example, the FOL intervention cost 
$34,900 per classroom serving 20 children on average. Thus, providing the intervention to 100 children would 
entail delivery of FOL to five classrooms for a total of $174,400. If the intervention results in one additional 
high school graduate among the 100 children served, the total benefit to society would be roughly two to three 
times the cost of FOL for the 100 children served. 

13 Schweinhart et al. (2005). 

14 Reynolds et al. (2011). 

15 Karoly (2012). 

16 Reynolds et al. (2011). 
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use across all participants by 0.2 year on average, the cost of the intervention would be repaid 
through the savings in educational costs alone. By way of comparison, the Chicago Child- 
Parent Centers program reduced special education use by 0.7 year per child by age 18, 17 so the 
FOL impact would not have to be as large to pay back the intervention cost. 

A similar hypothetical calculation can be made about the potential value of reducing 
grade repetition. The benefit-cost analysis for the Chicago Child-Parent Centers program 
estimated that the average per pupil cost for a year of regular education in Chicago public 
schools in 2007 dollars was $9,173, or $9,647 in 2010 dollars. Again, following the Chicago 
Child-Parent Centers program benefit-cost analysis and assuming that one less grade repeated 
saves educational costs at age 19, and discounting to age four, gives a savings of about $6,200 
for each year of reduction in grade retention. Thus, if FOL reduced grade retention on average by 
0.3 year, the cost of the intervention would be covered by the reduction in grade repetition. The 
Chicago Child-Parent Centers program reduced the percentage of students who were retained by 
15 percentage points as of age 15, 18 a smaller impact than what would be required ofFOL. 

These calculations for high school dropout, special education use, and grade retention 
have assumed that the costs of FOL would be recovered by improvements in a single education 
outcome. However, smaller impacts on some combination of these three education outcomes 
could sum to generate sufficient savings to offset the cost of the program. Thus, although FOL 
may not generate impacts on any one of these outcomes that would be as large as a more 
intensive intervention like the Chicago Child-Parent Centers program, even small changes in 
any one of these outcomes would generate hundreds of dollars in savings that could at least 
partially offset the per-child cost of the FOL intervention. 

Analyses of longitudinal data suggest that the current evidence base is too limited to as- 
sume that preschool children’s improved attention or similar noncognitive measures provide a 
strong causal link to later outcomes such as high school graduation, college attendance, or 
crime. 19 While there is some very promising work linking executive function, in particular, to 
measures of achievement in the early elementary grades, 20 it is not known whether those 
benefits translate into longer-term gains for children. Given the per-child cost of the FOL 
intervention relative to the cost to society of a high school dropout or the cost to the public 
sector of special education use or grade retention, even a relatively small change in any of these 
educational outcomes as a result of the FOL intervention could produce sufficient dollar 
benefits to cover the program costs and perhaps generate a positive economic return. 


17 Reynolds et al. (2011). 
ls Reynolds et al. (2011). 

19 See Duncan and Magnuson (2011) for a summary of these analyses. 
20 Blair and Razza (2007). 
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Chapter 6 

Summary of Findings and Lessons Learned 


This chapter summarizes the findings from the three rigorous studies that are examined in this 
report. The studies evaluated a multicomponent intervention that focused on strengthening 
teachers’ classroom management as a means to improve the social and emotional competence 
of preschool children. The intervention included 30 hours of teacher training, weekly in-class 
teacher consultations, stress management activities for teachers, and one-on-one clinical 
services for selected children. Building on the initial findings from CSRP in Chicago, MDRC 
then conducted two demonstrations of the CSRP program model, now called Foundations of 
Learning (FOL), in Newark and Chicago. 

The CSRP and FOL demonstrations provided an important opportunity to test the pro- 
gram model in two quite different urban preschool systems. The research team selected out- 
come measures to maximize what could be learned across the studies, given the available 
funding. This chapter ties together the findings and lessons from these studies, drawing on the 
analysis of program impacts in all three and the implementation and cost analyses in FOL- 
Newark and FOL-Chicago. 


The Context 

As described in Chapter 3, the New Jersey Supreme Court’s Abbott v. Burke decisions, requir- 
ing the State to increase funding for certain low-income school districts, positioned Newark 
ahead of most of the country in implementing structural changes to promote quality in pre- 
schools. For example, the Abbott mandates required smaller class sizes, a lead and assistant 
teacher in each classroom, higher teacher salaries, and lead teachers with a minimum credential 
of a bachelor’s degree. The resources that were available to the Chicago preschool programs, 
which were all part of Head Start in this study, were more typical of urban districts, as reflected, 
for example, in the larger class sizes than in Newark and reports of staffing shortages. 

Given these differences, it was thought that the Chicago classrooms might have a 
greater need for, and potential benefit from, the FOL intervention. On the other hand, it was 
also possible that the resource limitations in Chicago might compromise the ability to fully 
implement FOL. 
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Key Implementation, Impact, and Cost Findings 

As described in Chapter 3, while the key components of the intervention were implemented in 
both Newark and Chicago, the resource limitations in FOL-Chicago did appear to influence 
certain aspects of program implementation there. Chapter 4 showed that the FOL intervention 
nevertheless had a positive impact on key targeted outcomes in both Newark and Chicago, and 
Chapter 5 presented data related to the costs of implementing FOL in both cities. 

• The FOL intervention components were delivered at relatively high lev- 
els of dosage and quality, as rated by teachers in both Newark and Chi- 
cago. The shortage in classroom resources and staffing in Chicago did, 
however, appear to affect the focus of in-class consultation there. 

The CSRP and two FOL studies demonstrate that the components of the intervention 
can be delivered in diverse implementation settings and that teachers in both cities considered 
FOL to be a worthwhile and high-quality program. As discussed in Chapter 3, attendance at the 
teacher training was strong in both Newark and Chicago, and teachers in both cities expressed 
high levels of satisfaction with the training. 

While the overall amount of in-classroom consultation was similar in Newark and Chi- 
cago, the focus of the consultation differed somewhat across the sites. The Clinical Classroom 
Consultants (CCCs) and teachers in Newark were engaged in activities that were related to the 
implementation of The Incredible Years strategies consistently across classrooms. However, in 
Chicago, consultation was more varied, with some CCCs focusing more on assisting teachers 
with basic classroom tasks (such as helping children during lunch time, monitoring recess, and 
cleaning in the classroom) rather than on consultation directly related to implementation of the 
classroom management strategies. This perhaps reflected more limited staffing of preschool 
programs in Chicago compared with Newark. 

Despite these differences in implementation, differences in impacts between the two 
sites were not very different with respect to the child outcomes for which infomiation was 
collected in both Newark and Chicago. 

• CSRP and FOL improved teachers’ positive classroom management, 
specifically in areas that were directly targeted by the intervention. In 
addition, some evidence suggests that program group teachers’ positive 
classroom management was sustained one year later in FOL-Newark. 

Well-attended and well-received trainings, along with the consultation component of 
the model, led to improved teachers’ practices that the intervention targeted directly. Specifical- 
ly, trained observers found that, compared with control group teachers, the teachers who 
received the intervention had more warm and positive interactions and fewer sarcastic and 
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angry interactions with children and managed problem behavior more effectively when it 
occurred. In addition, the magnitude of these impacts is relatively large. These were all aspects 
of practice that were directly targeted by the training and consultation that teachers received. 
Moreover, teachers who received the intervention in Newark were observed to be still using 
selected components of the classroom strategies a full year after FOL implementation ended and 
CCCs were no longer working with them. This suggests that teachers continued to accept at 
least portions of The Incredible Years approach because it was both a good fit and feasible to 
implement even with only the initial year of in-classroom support. 

• Likely because of teachers’ improved classroom management, FOL 
classrooms had greater amounts of instructional time. 

In FOL-Newark, where this issue was examined, fewer disruptions and more orderly 
transitions appeared to free up additional classroom instructional time, with FOL classrooms 
having an average of 10 more minutes of instruction out of a 120-nrinute observation period. 
If this was representative of gains that could be achieved every school day, it would translate 
to 50 minutes more instruction per week, or a week’s more instruction over a school year. 
These improvements are consistent with a central hypothesis of the demonstration: that 
children with challenging behaviors may divert teachers from providing instruction to all 
children, and improving teachers’ classroom management skills can reduce unproductive time 
in preschool classrooms. 

• Conflict decreased among children in the intervention classrooms. In 
addition, FOL improved children’s approaches to learning and execu- 
tive function s kil ls. 

There is strong evidence that the FOL intervention improved the two child outcomes 
that it targeted most directly: children’s problem behavior and approaches to learning. With 
respect to problem behavior, the intervention reduced children’s negative interactions with their 
peers. With respect to approaches to learning, children were observed to be more engaged in the 
classroom and tasks of learning. The children also did better on tests of executive function, a 
competence that underlies approaches to learning through a combination of attention, inhibitory 
control, and short-term memory skills. All of these effects were moderate in size — smaller 
than the effects of the most widely touted programs (like Perry Preschool) but larger than the 
effects of large-scale evaluations (like Head Start). 

• The study provided no clear evidence that FOL improved children’s 
early literacy and mathematics s kil ls. 

During the planning of the demonstrations, it was hypothesized that improving young 
children’s social and emotional competence would be both an important outcome in its own 
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right and a pathway to improved academic achievement. This premise was based on a body of 
research suggesting that children’s behavior problems and challenges that make it difficult to 
regulate their behavior are linked with long-tenn difficulties in schooling. 1 

However, the positive impacts that were found in FOL-Newark and FOL-Chicago did 
not consistently translate into improvement in early academic skills. CSRP showed some 
initial evidence of improved early academic skills as a result of the intervention, but those 
effects were not corroborated by findings in the FOL sites or by the pooled analysis across 
CSRP and FOL. This absence of a clear link between social and emotional development and 
achievement outcomes is consistent with earlier work suggesting that measures of behavior 
problems and social skills, and to a lesser extent attention, may be weak predictors of later 
reading and math achievement. 2 

• At approximately $1,750 per child, FOL represented a 14 percent in- 
crease in program costs in Newark and a 21 percent increase in Chi- 
cago. There is no evidence at this point that the benefits of FOL out- 
weigh its cost. 

The roughly $1,750 per-child cost of FOL, reflecting both the teacher training and in- 
classroom consultation, was much higher than the cost of The Incredible Years training alone; 3 
however, the costs are on a par with the cost of other programs that use a mental health consult- 
ant in the classroom on a routine basis. 4 Because of the limited data that are available on 
outcomes when the children from FOL classrooms went to elementary school and beyond, there 
is no evidence that the cost of the intervention is outweighed by its benefits. 

If further data on children were collected as they continue through school, some of the 
intervention’s greatest potential cost savings could come from its impact on special education 
placement and eventual high school completion for individual children. 5 Previous studies of 
preschool interventions have not found significant effects on special education placement until 
later in elementary school, 6 and determining impacts on high school completion would require 
many years of follow-up data. The cost of special education services in comparison with the 
FOL intervention is quite high, and those who complete high school have higher individual 
earnings compared with the future earnings of those who drop out. This has implications for the 


'Raver (2002). 

2 Duncan et al. (2007). 

3 The costs associated with Tlie Incredible Years training sessions were higher in FOL than in the training 
package offered by the curriculum developer because of the additional cost of travel for the trainers. 

4 Florida State University, Center for Prevention and Early Intervention Policy (2006). 

5 See Chapter 5 for hypothetical cost-benefit projections. 

6 Weikart, Bond, and McNeil (1978); Karoly et al. (1998). 
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larger economy , 7 as long-term impacts on either special education placement or high school 
completion for even a small number of children would result in major savings that far outweigh 
the intervention’s cost. 


Implications for Program Implementation, Public Policy, and 
Research 

Previous reports highlighted early lessons on the implementation and impacts of the CSRP and 
FOL-Newark interventions . 8 This final report on the FOL demonstration provides the oppor- 
tunity to identify additional lessons as well as potential next steps in a policy and research 
agenda to increase the impact of preschool programs for low-income children. 

• Low levels of institutional resources and supports as well as teachers’ 
varied receptivity to new strategies may present challenges to imple- 
menting classroom-based strategies. 

The analyses of implementation data from FOL illustrated differences between the fo- 
cus of consultation activities in the Newark and Chicago classrooms, indicating that there was 
variation in the extent to which CCCs were able to engage teachers in implementing the 
classroom management strategies. Qualitative analyses revealed that classroom resource and 
staffing constraints as well as a lack of support by center directors and principals could some- 
times be major barriers to implementation. Teachers and CCCs alike had difficulty focusing on 
improving classroom management strategies when classrooms were understaffed. CCCs also 
reported that teachers’ receptivity to the intervention varied depending on their openness to 
learning and trying new techniques and their perception of their own immediate needs for 
assistance with classroom management. 

Although FOL addressed teacher stress, application of new classroom management 
strategies may still be challenging for highly stressed teachers in understaffed classrooms. 
Moreover, center directors and school principals should be engaged in initial and ongoing 
conversations about implementation to ensure that they understand, and agree with, the key 
tenets of the model and will support its implementation in the classroom. 

• Consultation models would benefit from the inclusion of a comprehen- 
sive plan for supervision and support of the consultants. 


7 Aos et al. (2012). 

8 Lloyd, Bangser, and Parkes (2009); Morris et al. (2010). 
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Even though qualified consultants were hired in FOL, they needed ongoing support, 
supervision, and professional development to provide consistently high-quality services to 
teachers and children in the classroom. While qualitative analyses suggest that consultation 
services varied based on individual classroom and teacher characteristics, the data also suggest 
that consultants found it difficult to provide critical feedback to teachers, an activity theorized 
to facilitate teacher change. Supervision of consultation staff could be strengthened by fre- 
quently monitoring the content of their consultation sessions to ensure that teachers are 
receiving the type of feedback that will facilitate changes in classroom practices that are 
aligned with the model. 

• More research is needed to understand the mechanisms through which 
consultation contributes to improvements in teacher practice. 

Multiple studies on the effectiveness of classroom consultation (also referred to in the 
literature as “coaching”) have demonstrated positive effects on teacher practice, over and above 
any effects realized from training alone. Not only do teacher practices change, but outcomes for 
children also trend in a positive direction . 9 The “value added” of consultation goes beyond 
simply an extra pair of hands in the classroom. For example, the positive impacts on teachers’ 
practice and children’s outcomes that were observed in CSRP occurred in the context of control 
classrooms that had a teacher’s aide who worked for the same amount of time as the consultant 
in the program classrooms. The question, however, is what dosage and combination of consulta- 
tion activities is most appropriate for classroom consultation models. 

Research on the appropriate dosage for consultation is inconclusive . 10 Findings from the 
FOL studies suggest that the level of intensity (six to seven hours per day once per week) and/or 
the duration of consultation (a full academic year) in this model may not be necessary. Impacts 
for children in Chicago were on a par with those in FOL-Newark despite the lower level of 
consultation activities that were directly related to The Incredible Years training content in 
FOL-Chicago. Perhaps because The Incredible Years strategies were relatively simple to 
implement, minimal use of proactive coaching strategies was sufficient to change teachers’ 
behavior. Important topics for future research include understanding the level and type of in- 
class consultation that is needed for relatively straightforward models such as The Incredible 
Years as well as for more complicated models. 

• Although an intervention such as FOL can improve children’s social 
and emotional competence, this is only one aspect of preschool quality. 


9 Joyce and Showers (2002); Landry, Anthony, Swank, and Monseque -Bailey (2009); Pianta et al. (2008). 

10 Wasik, Mattera, Lloyd, and Boiler (forthcoming). 
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Improved classroom management and the resulting changes in children’s behavior freed 
up more time for instruction in FOL classrooms. In addition, children in these classrooms might 
well have been ready and able to take advantage of the increased instruction, since they had 
greater executive function skills of memory, attention, and inhibitory control. In this way, FOL 
set the preconditions for improved learning. However, focusing on children’s behavior alone 
does not appear to be enough. It is possible that children did not benefit academically from the 
FOL intervention because the teachers were not trained sufficiently in the literacy and mathe- 
matics curricula and instructional approaches that would have enabled them to take advantage 
of the increased instructional time. Therefore, FOL can be adopted by preschool programs that 
are seeking to improve classroom management and children’s social-emotional competence and 
increase the amount of instructional time. However, in order to improve children’s academic 
skills in ways that carry over into elementary school, FOL may need to be augmented with an 
evidence-based literacy/language and/or mathematics program. 

• Interventions that focus on increasing preschool children’s competen- 
cies may need to be reinforced by continued support in kindergarten 
and beyond. 

The long-term impacts of preschool interventions may differ depending on the class- 
room environments and quality of instruction to which children are exposed once they complete 
their pre-kindergarten year. While the hope is that improvements in social and emotional 
competence will extend beyond the preschool year, kindergarten often represents a challenging 
transition for children, with higher student-teacher ratios, less attention on supporting a positive 
emotional and social climate, and greater expectations of independence and discipline. There- 
fore, it may be important to link the techniques that are learned in an intervention model like 
FOL with training for kindergarten and other elementary school teachers in order to reinforce 
the social and emotional competence that children develop during preschool. 

• Longer-term follow-up of children will be the best mechanism to deter- 
mine whether preschool interventions are cost-effective. 

At a minimum, follow-up data into third grade would provide a clearer picture of the 
implications of social-emotional and other interventions for special education placement. In 
addition, measures of academic performance in the third and fourth grades are more predictive 
of long-term educational outcomes such as high school completion than are such measures in 
earlier grades . 11 Long-term studies of preschool interventions suggest the added value of 
collecting information on a range of child outcomes. Interestingly, impacts of some of the best- 
known preschool interventions, such as Perry Preschool, appear to fade with regard to test 


"Duncan et al. (2007). 
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scores only to reemerge in adult outcomes . 12 Long-tenn follow-up is therefore important to 
determine whether there are critical domains of young children’s development that lead to 
sustained effects on later outcomes. 

In conclusion, realizing the full potential of preschool education will require a greater 
understanding of which programs and educational strategies can most effectively boost long- 
tenn outcomes for low-income children. It is important for practitioners to think about the goals 
for their program within the context of their organization’s capacity. The CSRP and FOL 
interventions present one approach to improving an important aspect of preschool quality. Other 
evaluations — including Head Start CARES (Classroom-based Approaches and Resources for 
Emotion and Social skill promotion), an early childhood education (preschool) research 
demonstration that operated in Head Start centers across the country — will continue to Ilirther 
the field’s understanding of children’s social and emotional competence, including how to 
achieve longer-term positive impacts. Other research is also seeking to understand the link 
between early academic skills and long-tenn school achievement and adult outcomes. One such 
study at MDRC will focus on the impact of early mathematics skills, and other studies by 
leading researchers are also under way. This collective body of work on preschool interventions 
should provide answers about how to produce high-quality, effective preschool programs. 


12 Weikart, Bond, and McNeil (1978); Karoly (2012). 
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Appendix A 

Baseline Characteristics of Students 




The Foundations of Learning Demonstration 
Appendix Table A.l 

Baseline Characteristics of Students in Research Sample, Newark 


Characteristic 

Program 

Group 

Control 

Group 

Female (%) 

48.6 

48.0 

Age a (years) 

4.4 

4.3 * 

Race/ethnicity b (%) 

African-American, not Flispanic 

42.2 

43.7 

White, not Hispanic 

9.5 

9.1 

Hispanic 

35.8 

34.4 

Other 

0.4 

1.2 

Number of government benefits received in household 

1.4 

1.4 

3 or more children in the household (%) 

33.1 

30.4 

Single-parent household (%) 

47.8 

50.0 

Primary language spoken at home is Spanish (%) 

18.2 

17.5 

Sample size 

319 

304 


SOURCES: MDRC calculations from responses to teacher and parent surveys. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

Means are adjusted for random assignment block but not for nesting of students within 
classrooms. 

a Age at start of school year, September 2007. 

b Means do not sum to 100 percent because the values represent adjusted means, and 
race/ethnicity is not available for all students. 
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The Foundations of Learning Demonstration 
Appendix Table A.2 

Baseline Characteristics of Students in Research Sample, Chicago 


Characteristic 

Mean 

Program Control 
Group Group 

Female (%) 

54.7 

47.5 * 

Age a (years) 

4.1 

4.1 

Race/cthnicity b (%) 

Black/ African- American, not Hispanic 

67.7 

72.9 

Hispanic 

30.1 

26.4 

Other 

2.3 

0.8 

Child's family receives TANF (%) 

7.9 

8.1 

Child's family income is at or below 50% of 
poverty line (%) 

44.7 

46.0 

Single-parent household (%) 

92.0 

91.0 

Primary language spoken at home is Spanish (%) 

29.7 

23.9 

Sample size 

298 

282 


SOURCES: MDRC calculations from responses to teacher and parent surveys. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

Means are adjusted for random assignment block but not for nesting of students within 
classrooms. 

a Age at start of school year, September 2008. 

b Means do not sum to 100 percent because the values represent adjusted means, and 
race/ethnicity is not available for all students. 
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Appendix B 

Supplemental Material for Chapters 2 and 3 




Data were obtained from the following sources for the Foundations of Learning (FOL) 
implementation study: 

• Attendance Logs. Measures of program dosage were collected using 
attendance logs from (1) teacher attendance at the training sessions; (2) 
Clinical Classroom Consultant (CCC) attendance in the classroom; and (3) 
teacher attendance in the classroom. 

• Service Provision Forms (SPFs). The CCCs completed the SPFs (shown in 
Appendix Figure B.l) on a weekly basis. The SPFs provided a variety of 
infonnation on the CCCs’ role in the classroom, the order of treatment 
implementation, the type of challenging behaviors that children were 
exhibiting in the classroom, the ways that teachers responded to these 
behaviors, goals for the upcoming week, and whether the intervention 
strategies as implemented by teachers were effective for the current week. 

The fonns also provided insight into the presence of behavior problems, the 
presence of modeling and consulting (as reported by CCCs), and the use of 
the intervention strategies in the CCCs’ absence (as reported by teachers to 
CCCs). In addition, the forms provided a snapshot of what the CCCs were 
doing in the classroom beyond consultation with teachers and children, such 
as assisting with meals and recess, collaborating with other school staff, and 
meeting with parents. This was useful for understanding the nature of 
program implementation and the amount of time the CCCs were present at 
the site. 

• Teacher Evaluations of Training. At the end of each training session, lead 
and assistant teachers at the FOL program sites were asked to complete 
evaluations of the training session. These evaluations measured teacher 
satisfaction with the training component of the intervention as well as the 
extent to which they perceived the training to be useful and relevant to their 
work in the classroom. 

• Teacher Evaluative Reports of CCCs. Lead teachers were asked to 
complete evaluations that reflected their experience and satisfaction with 
the CCCs. Teachers also documented whether or not they experienced the 
CCC services as useful and valuable to them and their students. 

• Stress Management Evaluations and Stress Management Workshop 
Surveys. Administered before and after the stress management workshops to 
lead and assistant teachers at FOL program sites, these surveys identified 
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sources of teacher stress and determined the extent to which these workshops 
helped to mitigate teacher stress. 

• Focus Group and Individual Interviews with Lead and Assistant 
Teachers. Interviews with lead and assistant teachers at FOL program sites 
were conducted to gain an understanding of the daily experiences and 
problems teachers in the classroom faced. In addition, the interviews 
provided an opportunity to discover the prevalence of disruptive and 
challenging behaviors in the classroom, the strategies teachers used to handle 
them, the services available to assist with this process, and how useful and/or 
successful teachers found them to be. Finally, teachers were asked to share 
their ideas about what might be done to improve the quality of services 
available to them and their students as part of the FOL intervention and 
within the school district at large. 

• Intervention and Programmatic Research Notes. These data included a 
variety of observational notes written by program staff during and after site 
visits, and notes from weekly CCC intervention fidelity meetings. 

• Individualized Child Treatment Plans and One-on-One Reports. These 
fonns provided data on the dosage of services received by children who were 
getting individualized consultation, the types of interventions conducted, and 
the number of referrals made once the consultation process was completed. 

• Focus Group with the Preschool Intervention and Referral Team 
(PIRT) and the Head Start Mental Health Consultants (MHCs). The 

focus group data from the PIRT and MHCs in Newark illuminated their 
perceptions about the FOL intervention implementation processes, provided 
insight into the extent to which the PIRT and MHCs coordinated with CCCs, 
and identified challenges associated with the collaboration effort. 
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The Foundations of Learning Demonstration 
Appendix Table B.l 

Early Childhood Field-Based Professionals in Newark and Chicago 


Site 

Title 

Site Category 

Frequency of Site -based Work 

Responsibilities 

Newark 

Preschool Intervention 
and Referral Team 
(P1RT) worker 

Public school; 
community-based 
organization (CBO) 

Weekly in public schools; varies in CBOs 

Child observations; child intervention 
plans; universal intervention programs 3 


Resource teacher 

Public school 

Every 1 to 2 weeks 

Refer children for observations by PIRT ; 
train, support, and monitor teacher 
implementation of curriculum; b on-site 
teacher support in wide range of preschool 
instruction topics 


Mental health 
consultant 

Flead Start 

On call/as needed 

Teacher training; child observations; 
referrals for children and families; parent 
trainings 

Chicago 

Mental health 
consultant 

Flead Start L 

On call/as needed 

Teacher training; child observations; direct 
child services; referrals for children and 
families; parent trainings 


Social worker 

Public school 
Flead Start 

On call/as needed 

Child observations; direct child services; 
referrals for children and families 


Master teacher 

Public school 

Monthly (approximately) 

On-site teacher support in wide range of 
preschool instruction topics d 


NOTES: 3 At the time of the study, Newark Public schools were using the Second Step curriculum. For more information, see www.cfchildren.org/second- 
step.aspx. 

b Creative Curriculum was the mandated curriculum in Newark preschool classrooms. For more information, see www.teachingstrategies.com/page/73756- 
creative-curriculum-system-preschool.cfm. 

c Chieago operates some Flead Start programs in public schools. 

d ln some cases, master teachers acted as lead classroom teachers when there was an unfilled position in a classroom. 



The Foundations of Learning Demonstration 
Table B.2 


Clinical Classroom Consultants' Demographic Information 


Characteristics 

Newark 

Chicago 

Female (%) 

85.7 

100 

Average age (years) 

32.4 

30 

Race and ethnicity (%) 

Caucasian 

28.6 

20 

Black/ African-American, not Hispanic 

14.3 

40 

Hispanic/Latino 

14.3 

20 

Other 1 

42.9 

20 

Bilingual 15 (%) 

Spanish 

14.3 

20 

Other 0 

14.3 

20 

Educational background (%) 

Social work 

57.1 

60 

Art therapy 

14.3 

0 

Psychology 11 

14.3 

20 

Counseling 

14.3 

20 

Professional experience in educational settings (%) 

0 to 4 years 

57.1 

60 

5 to 10 years 

42.9 

20 

Unknown 

0 

20 

Professional experience working with young children (%) 

0 to 4 years 

42.9 

60 

5 to 10 years 

57.1 

20 

Unknown 

0 

20 

Sample size 

7 

5 


SOURCE: MDRC calculations from Clinical Classroom Consultant (CCC) demographics surveys. 

NOTE: "Unknown" indicates that the CCC omitted an answer to the question. 

“Other includes "mixed" race/ethnicity or other race/ethnicity not listed. 
b Fluent in a second language in addition to English. 

“Other languages include Portuguese or language not specified by the CCC. 
d One CCC had a master's degree in psychology and education. 
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The Foundations of Learning Demonstration 
Appendix Figure B.l 
Sample Service Provision Form 


SPF # 

CCC Name 

Date / / 

Site/Room # 

Teacher: 


SELECT TEACHER 


Visit Report: Time in: 


Time out: 




1. Teacher received: □ 1 st training □ 2 nd training □ 3 rd training □ 4 th training □ booster 

Received in: SELECT SELECT SELECT SELECT SELECT 


□ all trainings 


2. Phase of model: 

□ 1 . pre -entry 

□ 2. entry 

□ 3. joining 


□ 4. assessment □ 7. direct services to children 

□ 5. consulting □ 8. referral 

□ 6. stress and burnout □ 9. termination 


3. Did you spend time on: 

□ a) Assisting with morning activities 
(i.e. reading materials, coloring, 

manipulatives) 

□ b) Assisting with meals 

□ c) Assisting with recess 

□ d) Assisting with classroom cleanup 

□ e) Assisting with field trips 

□ f) Attending school-related event/mtg. 

□ g) Reviewing training materials 

with teachers 

□ h) Coaching teachers on Webster-Stratton 

□ i) Classroom-based intervention 

□ j) Teacher stress reduction 

□ k) Referral/providing resources to 

teachers 


□ 1) Consulting with parents/guardians 

□ m) Referral/providing resources 

to parents 

□ n) Securing parental consent for 1-1 
intervention 

□ o) Direct child-level service 
provision 

□ p) Collaboration with center staff (i.e. 
director, cook) 

□ q) Collaboration with other on-site 
professionals (i.e. PRC, MHC, 
social worker, nurse, resource 
teacher) Who: 

□ r) Other: (explain) 


(continued) 
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Appendix Figure B.l (continued) 


4. Goal 

a. State the goal(s) for this visit 

b. Who identified goals? SELECT 

5. Techniques 

(1) Select the Webster-Stratton techniques used during this visit. 

(2) How did teacher rate the technique’s effectiveness? 

( 3 ) How would you rate the teacher’s success using the Webster-Stratton technique? 

a. Work with parents 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

b. Build positive relationships with students 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

c. Proactive Teaching 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

d. Promote Positive Behavior 

(teacher attention, encouragement and praise) 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

e. Using Incentives to Motivate Students 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

f. Managing Misbehavior (ignoring and redirecting) 

1) SELECT " 2) SELECT " 3) SELECT 

Please explain any additional observations: 

g. Managing Misbehavior (consequences) 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

h. Teaching students to problem solve 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

(continued) 
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Appendix Figure B.l (continued) 


i. Peer problems and friendship skills 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

j. Handling Emotions 

1) SELECT 2) SELECT 3) SELECT 

Please explain any additional observations: 

k. 1) Were there other techniques used? 

2) How did teacher rate the technique’s effectiveness? SELECT 

3) How would you rate the teacher’s success using the Webster-Stratton technique? SELECT 

6. Coaching Cycle 

Did you and the teacher attempt any steps in the coaching cycle? SELECT , SELECT, SELECT, 
SELECT, SELECT, SELECT 

Observation Notes: 

7. CCC/Teacher 

Discuss debriefing (any successes/issues/concems/challenges brought up by teachers) 


8. Goal Attainment 

Have the goal(s) of this visit been met? SELECT 
Explain: 

9. Plans for next visit 

1 . What goals does the teacher want to work on for the upcoming week? 

2. Please describe the intervention(s) that you and the teacher will use to meet these goals 

3. With the intervention, please describe the roles of the: 

Teacher: 

Teaching Assistant: 

CCC: 

PRC: 

MHC (if NPSC only): 

Resource teacher: 

Others: 

Clinical Classroom Consultant Signature: Date: / / 
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Appendix C 

Supplemental Material for Chapter 4 




Observed Ratings of Classroom Climate in FOL-Newark 

Trained observers in the FOL-Newark classrooms assessed the emotional and instructional 
climates of the classrooms using CLASS (Classroom Assessment Scoring System). 1 Average 
scores across observers and across four cycles of observations on a single day were computed for 
each of the following dimensions. In each dimension, scores ranged from 1 (low) to 7 (high). 

Positive Classroom Management 

• Positive climate reflects the overall emotional tone of the classroom. 
Observers evaluate the relationships between teachers and students, posi- 
tive affect or joint experiences of positive emotions (like group laughter), 
the degree of respect the teacher shows toward the students, and interac- 
tions among the students. 

• Negative climate gauges the severity and intensity (duration) of both 
teacher- and peer-expressed negativity in the classroom. Negativity refers 
to sarcasm, anger, and disrespect by the teacher and arguing, aggression, 
and bullying by students. 

• Teacher sensitivity measures the extent to which teachers comfort stu- 
dents, address students’ needs (academic and emotional), and create age- 
appropriate activities. Observers evaluate whether a teacher effectively ad- 
dresses student concerns and notices when they need assistance and 
whether the students feel comfortable enough to seek the teacher’s support. 

• Behavior management reflects a teacher’s ability to prevent student 
misbehavior by setting clear expectations, using effective praise to rein- 
force positive behavior, and proactively monitoring and redirecting pos- 
sible problems. Time taken to deal with behavioral issues and occurrenc- 
es of misbehavior are also factors. 

Quality of Language Instruction 

• Regard for student perspectives gauges the degree to which a teacher 
prepared activities that support students’ autonomy, expression of their 
ideas, taking on responsibilities, and socializing with peers. Teachers 
should be flexible during lessons by responding to student’s ideas. 


LaPai'o and Pianta (2003). 
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• Use of engaging teaching methods focuses on the teacher’s preparation 
of lessons and activities and the mode with which they deliver these ac- 
tivities. Teachers should use a variety of materials that promote explora- 
tion and actively question students. 

• Promoting understanding through conversation assesses how a 
teacher phrases and uses feedback to promote understanding of a topic 
and not simply to express whether or not students provide the correct an- 
swer. Teachers should prolong student-initiated conversations in order to 
get the student to gamer a greater understanding of a specific topic or 
give hints when necessary. 

• Encouragement of students’ language use measures the quality and 
amount of a teacher’s use of language to stimulate and encourage student 
conversation and responses. Teachers should frequently converse with 
children using open-ended questions about students’ responses and use 
of language that may be new to the students. 

Time Management 

• Management of classroom time measures the degree to which instruc- 
tional time is effectively managed, including minimal down time, 
through lesson planning, use of classroom routines, and the teacher’s 
ability to minimize disruptions. Management of classroom time also 
takes into account the amount of time spent making the transition be- 
tween activities. 

Observed Ratings of Child Behavior in FOL-Newark 

The team in Newark assessed the extent to which children were engaged in classroom 
activities, using the dimensions described below and the same l-to-7 scale that was used 
for the classroom observations. 2 

Problem Behavior 

• Teacher conflict measures the magnitude, consistency, and build-up of 
the child’s interactions with the teacher when they are characterized by 
tension, resistance, and negativity. The degree of verbal or physical ag- 


2 Downer et al. (2010); LaParo and Pianta (2003). 


116 



gression, noncompliance, negative affect, whining, and complaining di- 
rected toward the teacher is assessed. 

• Peer conflict gauges the magnitude, consistency, and build-up of the 
child’s interactions with his or her peers when they are characterized by 
tension, resistance and negativity. The degree of verbal or physical ag- 
gression, uncooperative behavior, negative affect, and whining and com- 
plaining, as well as whether the child sets children against one another, 
bullies, or is intrusive, are also evaluated. 

Approach to Learning 

• Task engagement measures the amount of time the child remained ac- 
tively engaged, focused, and on task. Also evaluated are the child’s abil- 
ity to follow directions, enthusiasm for classroom tasks and activities, 
and his or her willingness to voluntarily participate in these tasks. 

• Task self-reliance reflects the child’s ability to seek out learning oppor- 
tunities, work independently, and make the best use of classroom re- 
sources (including the teacher). The child’s persistence during frustrating 
tasks and his or her ability to link concepts to personal experience are al- 
so assessed. 

• Task behavior control measures the child’s ability to regulate his or her 
movement, physical activity, and verbalizations so that they match the 
expectations of the classroom activity. This dimension also focuses on 
children’s patience and ability to keep their hands to themselves and re- 
spect the personal space of others. 

Positive Social Behavior 

• Teacher communication reflects the child’s communication with all 
adults. This includes children’s ability to initiate and sustain conversa- 
tions and their overall use of language to convey needs and emotions as 
well as to communicate socially and share opinions with teachers. 

• Teacher positive engagement assesses the degree to which the child is 
emotionally connected to teachers. This can be expressed by seeking out 
and enjoying interactions with teachers and the child’s proximity to 
adults during tasks, along with his or her display of shared positive emo- 
tions (for example, laughing, smiling) with teachers. 
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• Peer communication reflects the child’s communication with his or her 
peers. Similar to teacher communication, this includes children’s ability 
to initiate, join, and sustain conversations and their overall use of lan- 
guage to convey needs and emotions as well as to communicate socially 
and share opinions with their peers. 

• Peer sociability measures the child’s experiences of positive emotions 
and behaviors with peers. The child’s desire to seek out and willingness 
to respond to interactions with his or her peers and the child’s popularity 
are evaluated. Observers also assess the child’s proximity to and conver- 
sation and eye contact with other children as well as his or her social 
awareness and ability to share. 

• Overall classroom student engagement is the only student-focused di- 
mension of the CLASS observations and captures students’ focus and 
participation during activities, across all children in the classroom. A dis- 
tinction is made between active and passive engagement and whether the 
engagement is sustained throughout the day. 


Teacher-Reported Outcomes for Children 

For all teacher-reported outcomes that are measured in this report, scores were computed if 75 
percent or more of the scale’s items were completed on the surveys. 

Problem Behavior 

• The Behavior Problem Index assessed children’s externalizing (such as 
aggression) and internalizing (such as withdrawal) problem behavior. 

The measure detennines the frequency, range, and type of childhood 
problem behavior for children age 4 and older. 3 Teachers are asked to 
rate each of 30 items according to how characteristic it was of the child 
on a 3-point scale (“not true,” “sometimes true,” “very true”). A total 
score was created by summing answers to the individual items. A 14- 
item Externalizing Problems subscale was created to assess the extent 
to which the child engaged in acting out and aggressive behaviors. A 14- 
item Internalizing Problems subscale assessed the extent to which the 
child was anxious or depressed. See Appendix Table C.l for factor load- 
ings and a list of all items. 


3 Zill and Petersen (1986). 
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• The Attention Problems subscale of the Caregiver-Teacher Report 
Form asked teachers to answer a series of questions about the child’s be- 
havior “now or within the past two months.” 4 The 3-point scale allowed 
teachers to report whether behaviors (for example, fidgets, wanders 
away, or fails to carry out assigned tasks) occur often (“very true”), 
sometimes (“somewhat true”), or never (“not true”). The subscale score 
is an average of nonmissing items and scores ranging from 0 to 1 8 in this 
sample. 

• Teachers’ perceptions of conflict and closeness with children were as- 
sessed with the Student-Teacher Relationship Scale (STRS). 5 The 
Teacher-Student Conflict and Teacher-Student Closeness subscales 
(included in Positive Social Behavior, below) include a total of 15 items 
that use a 5-point Likert-type rating scale to assess a teacher’s percep- 
tions of his or her relationship with a child, a child’s interactive behavior 
with the teacher, and a teacher’s beliefs about the child’s feelings toward 
the teacher. 

Positive Social Behavior 

• Teachers reported on a child’s positive behavior in the classroom using 
the Compliance with Teachers’ Directives (Compliance) and Social 
Competence (Social Competence and Sensitivity) subscales of the Posi- 
tive Behavior Scale (PBS). 6 The teacher responds on a 5-point scale, 
ranging from 1 (“never”) to 5 (“all of the time”) for the 11 -item Social 
Competence subscale (for example, gets along well with other children, 
shows concern for other people’s feelings); and the 8-item Compliance 
with Teachers’ Directives subscale (for example, thinks before he/she 
acts, usually does what I tell him/her). Reported scores are an average of 
nonmissing items. 

Approach to Learning 

• Teachers assessed children’s task engagement using the 16-item Work- 
Related Skills subscale of the Cooper-Farran Behavioral Ratings Scale 
(CFBRS). 7 The full measure is designed for use by teachers in assessing 


4 Achenbach (1997). 

5 Pianta (2001). 

6 Quint, Bos, and Polit (1997). 

7 Cooper and Farran (1991). 
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classroom behavior, with teachers asked to report on children’s behavior 
during classroom activities such as “designated work time,” and has been 
used extensively with preschool-age and kindergarten-age children. 
Showing good predictive validity for children’s later academic outcomes, 
the 7-point scale has descriptive phrases, which differ by item, to anchor 
responses to points. The scores shown for the Work-Related Skills sub- 
scale used in this report are an average, not standardized, score of non- 
missing items. 

Early Academic Skills 

• Academic skills were assessed using the Academic Rating Scale (ARS). 8 
The scale was designed to indirectly assess the process and products of 
children’s learning in school. The 21 items are divided into three sub- 
scales: General Knowledge (five items), Language and Literacy (nine 
items), and Mathematical Knowledge (seven items). Teachers compare 
the target child with his or her peers on a 5 -point scale, reflecting the de- 
gree (“not yet” to “shows proficiency”) to which a child demonstrates 
skills, knowledge, and behaviors. Total and subscale scores were calcu- 
lated as an average of nonmissing items. 


Direct Assessments of Children’s Approaches to Learning 

Executive Functioning and Self-Regulation Tasks 

• The Head-to-Toes measure consisted of 10 test trials in which children 
were asked to do the opposite of what the assessor did (for example, to 
touch his or her head when the assessor touched his or her toes). Children 
received two points for a correct response and one point if they initially 
started to respond incorrectly but self-corrected and ended in the correct 
position. Thus, scores ranged from 0 to 20, and in the FOL sample the 
average score was 5.5 (standard deviation = 6.8). 

• The Pencil Tap measure is an adaptation of Luria’s“peg tap” task. 9 The 
task asks the child to do the opposite of the assessor (for example, tap a 
pencil two times on a table when the assessor taps one time, or vice ver- 
sa), and involves familiarization and practice segments in addition to test 


8 National Center for Education Statistics (n.d.). 

9 Luria (1966); Diamond and Taylor (1996). 
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trials. Pencil Tap scores range from 0 percent correct to 100 percent cor- 
rect across 16 test trials (with children who do not proceed to test trials 
receiving a score of 0), and in the FOL sample the average score was 56 
percent correct (standard deviation = 36 percent). 

• The Gift Wrap measure was used to assess children’s ability to inhibit 
their behavior. 10 Children were asked to turn around, to close their eyes, 
and to not “peek” while the assessor noisily wrapped a gift that the chil- 
dren were told they would get after the “wrapping” was complete. Scores 
represent the time in seconds until children peek, ranging from 0 (peeked 
immediately) to 60 seconds (never peeked), and in the FOL sample the 
average score was 38.4 seconds (standard deviation = 24.2). 

Children’s Assessment-Related Behavior 

• The Preschool Self-Regulation Interviewer Assessment (PSRA) was 

used to measure children’s attentiveness, inhibitory control, positive en- 
gagement, and negative emotionality during the assessments that observ- 
ers administered to children. Assessors rated children on a scale of 1 to 4 
over 26 items, and exploratory factor analyses identified the dimensions 
of attention, inhibitory control (16 items), and positive engagement (6 
items). 11 See Appendix Table C.2 for factor loadings. 


Direct Assessment of Non-Targeted Child Outcomes 

Social Problem-Solving Skills 

• The problem-solving portion of the Challenging Situations Task (CST) 
was used to measure children’s behavioral responses to common peer- 
related conflict situations. 12 Children are presented with pictures that rep- 
resent four hypothetical peer situations (for example, “You are building a 
tower with blocks and Bobbi came and knocked your tower down”), and 
are asked what they would do in such a situation. Children are presented 
with four pictures of behavioral responses (representing “prosocial,” 
“aggressive,” “avoidant,” and “adult-dependent” responses) and asked to 


10 Kochanska et al. (1996). 

n A fourth factor, negative emotionality (four items), was also identified but was not evaluated for the im- 
pacts of FOL. 

12 Denham, Bouril, and Belouad (1994). 
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choose one. Frequency counts for each behavioral response were calcu- 
lated across the four situations, each with a possible range of 0 to 4. An 
adaptive problem-solving composite score was created by reverse-coding 
the aggressive and avoidant scores and adding these to the prosocial and 
adult-dependent scores, with scores ranging from 4 (more negative) to 12 
(more positive). 13 The FOL-Chicago sample had a mean score of 8.84 
and a standard deviation of 2.16. 

Early Academic Skills 

• The Letter-Word Identification and the Applied Problems subtests of 
the Woodcock Johnson Tests of Achievement III (WJ-HI) assessed chil- 
dren’s early math and early reading skills. 14 Raw scores were standard- 
ized using the WJ-III standardization software, which uses the child’s 
age and raw score to determine a standardized score based on population 
nonns. Standardized scores on WJ-III tests have a mean of 100 and a 
standard deviation of 15. 

• The Peabody Picture Vocabulary Test (PPVT) measures receptive 
(hearing) vocabulary for Standard American English and provides a 
quick estimate of verbal ability or scholastic aptitude. 15 Using the PPVT 
Form IIIA, children are shown groups of four pictures and are asked to 
identify the picture that corresponds to a word they are told. Standardized 
scores are calculated based on a child’s age and raw score. Standardized 
scores on PPVT tests have a mean of 100 and a standard deviation of 15. 


l3 Domitrovich, Cortes, and Greenberg (2007). 
14 McGrew, Schrank, and Woodcock (2007). 
15 Dunn and Dunn (1997). 
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The Foundations of Learning Demonstration 
Appendix Table C.l 


Items and Factor Loadings for the Behavior Problems Index Subscales 



Factor Loading 


Internalizing 

Externalizing 

Item in Total Scale 

Behavior 

Behavior 

Is rather high-strung and nervous 

0.4 


Is secretive, keeps things to himself/herself 

0.6 


Worries too much 

0.8 


Is too fearful or anxious 

0.6 


Is easily confused, seems to be in a fog 

0.4 


Feels worthless or inferior 

0.6 


Has difficulty getting his/her mind off certain thoughts 

0.5 


Is unhappy, sad, or depressed 

0.7 


Is withdrawn, does not get involved with others 

0.8 


Clings to adults 

0.6 


Cries too much 

0.6 


Demands a lot of attention 

0.5 


Is too dependent on others 

0.6 


Has sudden changes in mood or feeling 1 * 

0.3 

0.5 

Hangs around with kids who get in trouble 


0.6 

Cheats or tells lies 


0.7 

Argues too much 


0.8 

Has difficulty concentrating, cannot pay attention for long 


0.5 

Bullies or is cruel or mean to others 


0.8 

Is disobedient 


0.9 

Does not seem to feel sorry after he/she misbehaves 


0.8 

Has trouble getting along with other children 


0.7 

Is impulsive, or acts without thinking 


0.8 

Is not liked by other children 


0.5 

Is restless or overly active, cannot sit still 


0.7 

Is stubborn, sullen, or irritable 


0.7 

Has a very strong temper and loses it easily 


0.8 

Breaks things on purpose or deliberately destroys his/her own or another's things 

Feels others are out to get him/her 

Feels or complains that no one loves him/her 


0.7 

Cronbach coefficient alpha for scale 

0.9 

0.9 


SOURCE: Based on MDRC calculations from responses to teacher survey. 

NOTES: A promax rotation, two-factor structure was used to identify subscales. Only factor loadings greater than 
or equal to |.30| are shown, based on factor analysis. Factor loadings indicate items that were used to create the 
respective scales. Items were included on the factors on which they most highly loaded. 
a This item was grouped with the Internalizing Behavior subscale based on theory. 
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The Foundations of Learning Demonstration 
Appendix Table C.2 

Items and Factor Loadings for the PSRA Assessor Reported Subscales 



Factor Loading 



Attentiveness/ 

Positive 

Item in Total Scale 

Inhibitory Control Engagement 

Pays attention 

0.74 


Careful 

0.72 


Sustains concentration 

0.79 


Daydreams 

-0.62 


Distracted by sights 

-0.76 


Destructive with materials 

-0.63 


Thinks and plans 

0.83 


Refrains from touching 

0.81 


Does not interrupt 

0.82 


Difficulty waiting 

-0.83 


Remains in seat 

0.70 


Defiant 

-0.71 


Passively noncompliant 

-0.66 


Regulates arousal in self 

0.75 


Aggressiveness aggregate 0-3 

-0.35 


Cooperates and complies 
Alert and interactive 

0.52 

0.67 

Actively engages interviewer 


0.69 

Shows pleasure in accomplishment 


0.77 

Confident 


0.62 

Shows intense positivity 


0.76 

Shows frequent positivity 
Shows intense irritability 
Shows frequent irritability 
Shows intense apprehension 
Shows frequent apprehension 


0.75 

Cronbach coefficient alpha for scale 

0.94 

0.86 


SOURCE: Based on MDRC calculations from the interviewer assessment segment of the direct child 
assessment. 


NOTES: A varimax rotation, three-factor structure was used to identify subscales. Only factor 
loadings greater than or equal to |.30| are shown, based on factor analysis. Factor loadings indicate 
items that were used to create the respective scales. Items were included on the factors on which they 
most highly loaded. A third subscale was not used in this report. 
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The Foundations of Learning Demonstration 
Appendix Table C.3 


Chicago Program Impacts on Teacher-Reported Child Outcomes, 
Pre-Kindergarten Year 



Mean Score 





Program 

Control 

Difference 

Standard 

Effect 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Problem behavior 

Internalizing problems 

2.5 

1.5 

1.0 

0.6 

0.4 

Externalizing problems 

5.9 

5.0 

0.8 

1.3 

0.15 

Teacher-student conflict 

12.9 

13.2 

-0.3 

1.8 

-0.05 

Attention problems 

4.8 

4.2 

0.6 

1.7 

0.13 

Approaches to learning 

Work-related skills 

4.8 

4.9 

-0.1 

0.27 

-0.09 

Positive social behavior 

Compliance with teachers' directives 

3.7 

3.7 

-0.1 

0.21 

-0.07 

Social competence 

3.7 

3.8 

-0.1 

0.19 

-0.20 

Teacher-student closeness 

34.5 

34.6 

-0.1 

1.99 

-0.02 

Earlv academic skills 

Language and literacy skills 

2.9 

3.1 

-0.2 

0.37 

-0.12 

Math knowledge 

2.7 

2.8 

-0.2 

0.46 

-0.12 

Sample size 






Students 

266 

257 




Classrooms 

20 

20 





SOURCE: Based on MDRC calculations of teacher reports on children. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Regression-adjusted means control for random assignment status and pairs, site characteristics, and 
baseline child and teacher characteristics. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the 
control group. 

"Internalizing problems" and "Externalizing problems" refer to subscales of the Behavior Problems 
Index. "Teacher-student conflict" and "Teacher-student closeness" refer to the Student-Teacher 
Relationship Scale conflict and closeness subscales. "Attention problems" refers to the Caregiver-Teacher 
Report Form attention problems subscale. "Work-related skills" refers to a subscale of the Cooper-Farran 
Behavioral Rating Scales. "Compliance with teachers' directives" and "Social competence" refer to the 
Positive Behavior Scale subscales. "Language and literacy skills" and "Math knowledge" refer to the 
subscales of the Academic Rating Scale. 
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The Foundations of Learning Demonstration 
Appendix Table C.4 


ON 


Chicago Program Impacts on Observed and Teacher Ratings of Child Outcomes, Pre-Kindergarten Year, 

by Age Range of Child 

3 Years Old and Younger 4 Years Old and Older Subgroup 

Mean Score Mean Score Impact 

Program Control Difference Standard Effect Program Control Difference Standard Effect Difference 
Outcome Group Group (Impact) Error Size Group Group (Impact) Error Size (Sig.) 


Assessments 


Positive engagmenf* 

2.1 

1.9 

0.2 

0.16 

0.45 

2.1 

1.9 

0.2 

0.15 

0.31 

Attentiveness/inhibitory control 15 

2.1 

2.3 

-0.1 

0.21 

-0.19 

2.6 

2.5 

0.1 

0.12 

0.16 

Executive functioning 

Plead to Toes (gross motor) 0 

4.0 

2.2 

1.8 

1.62 

0.39 

8.1 

5.7 

2.4 

2.40 

0.36 

Pencil Tap (fine motor) 01 

0.4 

0.4 

0.0 

0.10 

0.07 

0.6 

0.6 

0.0 

0.11 

0.00 

Gift Wrap (measured in seconds) 

38.6 

29.3 

9.3 

7.23 

0.38 

44.3 

36.6 

7.7 

6.33 

0.33 

Social problem-solving skills 

Adaptive problem-solving 

8.6 

OO 
o o 

-0.2 

0.62 

-0.09 

OO 

OO 

8.9 

-0.2 

0.54 

-0.08 

Early academic skills 0 

Peabody Picture Vocabulary Test 

87.8 

80.6 

7.2 

7.09 

0.48 

84.3 

82.3 

2.0 

7.03 

0.13 

Woodcock Johnson 

Early math skills 

107.3 

96.2 

11.1 ** 

3.66 

0.81 

95.7 

94.0 

1.7 

3.72 

0.18 

Early language/literacy skills 

107.8 

102.5 

5.3 

4.17 

0.32 

99.4 

97.7 

1.6 

2.95 

0.15 


Teacher reports' 

Problem behaviors 


Internalizing problems 

2.0 

1.7 

0.3 

0.96 

0.19 

2.7 

1.3 

1.4 * 

0.71 

0.54 

Externalizing problems 

5.1 

4.8 

0.3 

2.20 

0.06 

6.0 

5.2 

0.7 

1.58 

0.13 

Teacher-student conflict 

9.9 

13.9 

-4.0 

3.42 

-0.62 

13.2 

13.2 

-0.1 

1.81 

-0.01 

Attention problems 

4.4 

5.4 

-1.0 

3.40 

-0.18 

4.0 

3.5 

0.5 

1.48 

0.11 

Approaches to learning 

Work-related skills 

4.6 

4.4 

0.2 

0.59 

0.14 

5.1 

5.2 

-0.1 

0.31 

-0.07 
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Appendix Table C.4 (continued) 




3 Years Old and Younger 



4 Years Old and Older 


Subgroup 


Mean Score 




Mean Score 




Impact 


Program 

Control 

Difference 

Standard 

Effect 

Program 

Control 

Difference 

Standard 

Effect 

Difference 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Group 

Group 

(Impact) 

Error 

Size 

(Sig.) 

Teacher reports 












Positive social behavior 












Compliance with teachers' directives 

3.7 

3.6 

0.1 

0.37 

0.18 

3.7 

3.8 

0.0 

0.24 

-0.06 


Social competence 

4.1 

3.6 

0.5 

0.40 

0.82 

3.8 

3.9 

-0.1 

0.23 

-0.19 


Teacher-student closeness 

37.6 

33.5 

4.1 

4.66 

0.75 

34.8 

34.9 

-0.1 

1.64 

-0.03 


Early academic skills 












Language and literacy skills 

2.2 

2.5 

-0.3 

0.68 

-0.25 

3.2 

3.4 

-0.2 

0.45 

-0.15 


Math knowledge 

1.9 

2.3 

-0.5 

0.71 

-0.49 

3.0 

3.1 

-0.1 

0.63 

-0.04 


Sample size 












Observations of students 

61 

54 




92 

97 





Teacher reports on students 

113 

92 




152 

165 





Classrooms 

20 

20 




20 

20 






SOURCES: Based on MDRC calculations of child assessments and teacher reports. 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. The effect size equals the impact divided by the 
standard deviation of the outcome measure for the control group. The H-statistic test was used to test for statistically significant differences in impact estimates 
across different subgroups. Statistical significance levels are indicated as follows: ff f = 1 percent; f f = 5 percent; f = 10 percent. 

Regression-adjusted means control for random assignment status and pairs, site characteristics, and baseline child and teacher characteristics. 

a This outcome is an average score of 6 items on a scale of 1 to 4. 

b This oucome represents an average score of 1 6 items on a scale of 1 to 4. 

c Seores on this measure may range from 0 to 20, with 0 representing no success and 20 representing success across the task's 10 trials. 
d Scores reflect the percentage of correct responses across the task's 16 trials. 

e All test scores are standardized, based on child's age and raw score, with a mean of 100 and a standard deviation of 15. 

f ”Intemalizing problems” and "Externalizing problems” refer to the subscales of the Behavior Problems Index. "Teacher- student conflict" and "Teacher- 
student closeness" refer to the Student-Teacher Relationship Scale conflict and closeness subscales. "Attention problems” refers to the Caregiver-Teacher 
Report Form attention problems subscale. "Work-related skills” refers to the Cooper-Farran Behavioral Rating Scales. "Compliance with teachers' directives” 
and "Social competence” refer to the Positive Behavior Scale compliance and social competence subscales. "Language and literacy skills” and "Math 
knowledge” refer to the subscales of the Academic Rating Scale. 



The Foundations of Learning Demonstration 
Appendix Table C.5 


Chicago Program Impacts on Observed and Teacher Ratings of Child Outcomes, Pre-Kindergarten Year, 

by Child's Level of Behavior Problem 


Low High Subgroup 

Mean Score Mean Score Impact 

Program Control Difference Standard Effect Program Control Difference Standard Effect Difference 
Outcome Group Group (Impact) Error Size Group Group (Impact) Error Size (Sig.) 


Assessments 


Positive engagmenf* 

2.1 

1.8 

0.30 

0.30 

0.50 

2.0 

2.0 

-0.01 

0.19 

-0.02 

Attentiveness/inhibitory control 13 

2.5 

2.6 

-0.01 

0.14 

-0.02 

2.4 

2.3 

0.13 

0.18 

0.23 

Executive functioning 

Head to Toes (gross motor) c 

8.6 

5.3 

3.23 

1.91 

0.46 

5.9 

3.8 

2.13 

2.22 

0.37 

Pencil Tap (fine motor) d 

0.7 

0.6 

0.13 

0.08 

0.37 

0.5 

0.6 

-0.14 

0.11 

-0.41 

Gift Wrap (measured in seconds) 

48.1 

39.2 

8.90 

5.55 

0.36 

38.7 

30.0 

8.75 

7.98 

0.36 

Social problem-solving skills 

Adaptive problem-solving 

8.9 

8.9 

-0.03 

0.60 

-0.02 

8.7 

9.0 

-0.34 

0.68 

-0.15 

Early academic skills 2 

Peabody Picture Vocabulary Test 

90.1 

83.4 

6.71 

6.11 

0.39 

81.3 

86.8 

-5.43 

6.14 

-0.47 

Woodcock Johnson 

Early math knowledge 

103.3 

95.1 

8.20 * 

3.88 

0.58 

96.6 

93.1 

3.50 

2.63 

0.47 

Early language/literacy skills 

103.9 

102.5 

1.41 

3.47 

0.09 

98.6 

96.6 

2.01 

3.28 

0.21 

Teacher reDorts* 

Problem behaviors 

Internalizing problems 

1.1 

1.1 

-0.06 

0.47 

-0.03 

4.0 

2.4 

1.64 

1.29 

0.57 

Externalizing problems 

2.2 

2.7 

-0.54 

1.18 

-0.13 

10.0 

8.5 

1.49 

2.89 

0.23 

Teacher-student conflict 

9.6 

10.6 

-1.08 

1.80 

-0.21 

16.3 

17.8 

-1.50 

2.75 

-0.22 

Attention problems 

2.5 

3.1 

-0.66 

2.06 

-0.13 

5.5 

5.8 

-0.27 

2.03 

-0.06 

Approaches to learning 

Work-related skills 

5.3 

5.3 

-0.01 

0.32 

-0.01 

4.3 

4.5 

-0.19 

0.43 

-0.17 
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Appendix Table C.5 (continued) 





Low 





High 



Subgroup 


Mean Score 




Mean Score 




Impact 


Program 

Control 

Difference 

Standard Effect 

Program Control Difference 

Standard Effect Difference 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Group 

Group 

(Impact) 

Error 

Size 

(Sig.) 

Teacher reDorts 

Positive social behavior 












Compliance with teachers' directives 

4.0 

3.9 

0.08 

0.21 

0.13 

3.4 

3.5 

-0.08 

0.27 

-0.12 


Social competence 

4.0 

4.0 

-0.04 

0.23 

-0.08 

3.5 

3.7 

-0.11 

0.25 

-0.20 


Teacher-student closeness 

35.8 

35.1 

0.71 

1.99 

0.14 

33.2 

34.3 

-1.16 

2.51 

-0.22 


Early academic skills 












Language and literacy skills 

3.2 

3.1 

0.18 

0.50 

0.13 

2.7 

3.2 

-0.52 

0.43 

-0.45 


Math knowledge 

2.9 

3.0 

-0.03 

0.56 

-0.02 

2.5 

3.0 

-0.49 

0.44 

-0.40 


Sample size 












Observations of students 

70 

61 




77 

69 





Teacher reports on students 

127 

103 




119 

97 





Classrooms 

20 

20 




20 

20 






SOURCES: Based on MDRC calculations of child assessments and teacher reports. 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. The effect size equals the impact divided by the 
standard deviation of the outcome measure for the control group. The H-statistic test was used to test for statistically significant differences in impact 
estimates across different subgroups. Statistical significance levels are indicated as follows: fff = 1 percent; f f = 5 percent; t = 10 percent. 
Regression-adjusted means control for random assignment status and pairs, site characteristics, and baseline child and teacher characteristics. 
a This outcome is an average score of 6 items on a scale of 1 to 4. 
b This oucome represents an average score of 16 items on a scale of 1 to 4. 

c Seores on this measure may range from 0 to 20, with 0 representing no success and 20 representing success across the task's 10 trials. 
d Scores reflect the percentage of correct responses across the task's 16 trials. 

e All test scores are standardized, based on child's age and raw score, with a mean of 100 and a standard deviation of 15. 

f " Internalizing problems" and "Externalizing problems" refer to the subscales of the Behavior Problems Index. "Teacher-student conflict" and "Teacher- 
student closeness" refer to the Student-Teacher Relationship Scale conflict and closeness subscales. "Attention problems" refers to the Caregiver-Teacher 
Report Form attention problems subscale. "Work-related skills" refers to the Cooper -Farran Behavioral Rating Scales. "Compliance with teachers' 
directives" and "Social competence" refer to the Positive Behavior Scale compliance and social competence subscales. "Language and literacy skills" and 
"Math knowledge" refer to the subscales of the Academic Rating Scale. 
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Appendix Table C.6 

Chicago Program Impacts on Observed and Teacher Ratings of Child Outcomes, 

Pre-Kindergarten Year, by Child's Gender 

Girl Boy Subgroup 

Mean Score Mean Score Impact 

Program Control Difference Standard Effect Program Control Difference Standard Effect Difference 


Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Group 

Group 

(Impact) 

Error 

Size 

Assessments 

Positive engagmenf* 

2.2 

1.9 

0.4 

0.21 

0.70 

1.9 

1.8 

0.1 

0.15 

0.22 

Attentiveness/inhibitory control 1 * 

2.4 

2.5 

- 0.1 

0.15 

- 0.19 

2.5 

2.3 

0.2 

0.16 

0.32 

Executive functioning 

Head to Toes (gross motor) c 

8.0 

2.7 

5.3 ** 

1.65 

0.90 

6.0 

5.3 

0.7 

2.00 

0.10 

Pencil Tap (fine motor) d 

0.5 

0.6 

- 0.1 

0.10 

- 0.17 

0.6 

0.5 

0.1 

0.08 

0.25 

Gift Wrap (measured in seconds) 

48.9 

39.1 

9.8 

6.86 

0.41 

35.6 

30.0 

5.6 

5.75 

0.22 

Social problem-solving skills 

Adaptive problem-solving 

9.1 

8.7 

0.4 

0.61 

0.17 

8.7 

OO 

oo 

- 0.1 

0.51 

- 0.05 

Early academic skills'* 

Peabody Picture Vocabulary Test 

87.2 

77.7 

9.5 * 

4.77 

0.57 

84.2 

84.1 

0.1 

6.48 

0.01 

Woodcock Johnson 

Early math skills 

99.8 

92.9 

6.8 * 

3.29 

0.64 

99.4 

94.6 

4.8 

3.19 

0.41 

Early language/literacy skills 

102.8 

98.5 

4.3 

3.43 

0.37 

100.0 

99.0 

1.0 

4.02 

0.07 

Teacher reDorts 1 

Problem behaviors 

Internalizing problems 

2.5 

1.2 

1.3 * 

0.67 

0.66 

2.7 

1.8 

0.9 

0.87 

0.35 

Externalizing problems 

5.2 

3.8 

1.4 

1.27 

0.31 

6.3 

6.8 

- 0.5 

2.10 

- 0.07 

Teacher-student conflict 

11.7 

12.2 

- 0.5 

2.12 

- 0.09 

13.9 

14.5 

- 0.6 

2.22 

- 0.09 

Attention problems 

3.9 

3.1 

0.8 

1.97 

0.16 

5.3 

5.1 

0.3 

1.69 

0.05 

Approaches to learning 

Work-related skills 

5.1 

5.2 

- 0.1 

0.30 

- 0.05 

4.4 

4.5 

- 0.1 

0.34 

- 0.10 

Positive social behavior 

Compliance with teachers' directives 

3.8 

3.9 

- 0.1 

0.26 

- 0.12 

3.6 

3.5 

0.1 

0.23 

0.08 

Social competence 

3.9 

4.0 

- 0.1 

0.21 

- 0.29 

3.6 

3.6 

0.0 

0.23 

- 0.02 

Teacher-student closeness 

36.5 

35.6 

1.0 

2.02 

0.20 

32.9 

33.8 

- 0.9 

2.37 

- 0.17 
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Girl 





Boy 



Subgroup 


Mean Score 




Mean Score 




Impact 


Program Control Difference 

Standard Effect 

Program Control Difference 

Standard 

Effect 

Difference 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Group 

Group 

(Impact) 

Error 

Size 

(Sig.) 

Teacher reDorts 












Early academic skills 












Language and literacy skills 

3.2 

3.1 

0.1 

0.42 

0.08 

2.6 

3.1 

-0.5 

0.40 

-0.37 


Math knowledge 

2.9 

2.9 

0.0 

0.54 

-0.04 

2.4 

2.8 

-0.4 

0.41 

-0.33 


Sample size 












Observations of students 

76 

68 




78 

81 





Teacher reports on students 

140 

122 




124 

133 





Classrooms 

20 

20 




20 

20 






SOURCES: Based on MDRC calculations of child assessments and teacher reports. 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. The effect size equals the impact divided by the 
standard deviation of the outcome measure for the control group. The H-statistic test was used to test for statistically significant differences in impact 
estimates across different subgroups. Statistical significance levels are indicated as follows: fff = 1 percent; tt = 5 percent; f = 10 percent. 
Regression-adjusted means control for random assignment status and pairs, site characteristics, and baseline child and teacher characteristics. 
a This outcome is an average score of 6 items on a scale of 1 to 4. 
b This oucome represents an average score of 16 items on a scale of 1 to 4. 

c Seores on this measure may range from 0 to 20, with 0 representing no success and 20 representing success across the task's 10 trials. 
d Scores reflect the percentage of correct responses across the task's 16 trials. 

e All test scores are standardized, based on child's age and raw score, with a mean of 100 and a standard deviation of 15. 

^Internalizing problems" and "Externalizing problems" refer to the subscales of the Behavior Problems Index. "Teacher-student conflict" and "Teacher- 
student closeness" refer to the Student-Teacher Relationship Scale conflict and closeness subscales. "Attention problems" refers to the Caregiver-Teacher 
Report Form attention problems subscale. "Work-related skills" refers to the Cooper-Farran Behavioral Rating Scales. "Compliance with teachers' 
directives" and "Social competence" refer to the Positive Behavior Scale compliance and social competence subscales. "Language and literacy skills" and 
"Math knowledge" refer to the subscales of the Academic Rating Scale. 
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Newark Program Impacts on Observed and Teacher Ratings of Child Outcomes, Pre-Kindergarten Year, 

by Race/Ethnicity of Child 


African-American, Not Hispanic Hispanic Subgroup 


Mean Score Mean Score Impact 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Effect 

Size 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Effect 

Size 

Difference 

(Sig.) 

Observations 3 

Problem behavior 

Teacher conflict 

1.3 

1.5 

-0.2 

0.1 

-0.43 

1.2 

1.5 

-0.3 

0.2 

-0.37 


Peer conflict 

1.4 

1.8 

-0.4 ** 

0.2 

-0.46 

1.4 

1.7 

-0.3 * 

0.2 

-0.48 


Positive social behavior 

Teacher communication 

2.2 

2.2 

0.1 

0.2 

0.07 

2.3 

2.5 

-0.2 

0.2 

-0.27 


Teacher positive engagement 

3.4 

3.3 

0.1 

0.3 

0.15 

2.9 

3.6 

q y ** 

0.2 

-0.86 

tt 

Peer communication 

2.8 

2.7 

0.1 

0.2 

0.11 

2.5 

2.9 

-0.3 

0.3 

-0.37 


Peer sociability 

3.7 

3.7 

0.0 

0.2 

0.01 

3.2 

3.7 

-0.6 ** 

0.2 

-0.64 

t 

Peer assertiveness 

2.4 

2.3 

0.1 

0.2 

0.07 

1.9 

2.6 

-0.7 * 

0.3 

-0.69 

t 

Approach to learning 

Task engagement 

4.9 

4.7 

0.2 

0.2 

0.21 

5.0 

4.8 

0.2 

0.2 

0.21 


Task self-reliance 

3.1 

3.1 

0.0 

0.3 

0.00 

3.1 

3.4 

-0.3 

0.3 

-0.29 


Task behavior control 

5.4 

5.0 

0.4 

0.3 

0.41 

5.3 

5.1 

0.2 

0.3 

0.22 


Teacher reDorts b 

Problem behavior 

Internalizing problems 

3.5 

2.4 

1.2 

0.8 

0.38 

2.4 

2.8 

-0.3 

1.2 

-0.08 


Externalizing problems 

4.6 

4.1 

0.5 

1.2 

0.09 

4.2 

4.2 

0.0 

0.9 

0.00 


Teacher-student conflict 

12.1 

11.6 

0.5 

1.5 

0.08 

12.4 

12.9 

-0.5 

1.2 

-0.09 


Positive social behavior 

Social competence 

4.0 

3.9 

0.1 

0.1 

0.15 

4.0 

4.0 

-0.1 

0.2 

-0.10 


Teacher-student closeness 

34.8 

36.2 

-1.4 

1.3 

-0.23 

34.5 

35.6 

-1.1 

1.1 

-0.30 
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African-American, Not Hispanic 




Hispanic 



Subgroup 


Mean Score 




Mean Score 




Impact 


Program 

Control 

Difference 

Standard 

Effect 

Program 

Control 

Difference 

Standard 

Effect 

Difference 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Group 

Group 

(Impact) 

Error 

Size 

(Sig.) 

Teacher reports (continued) 1 ’ 

Approach to learning 












Work-related skills 

4.7 

4.7 

-0.1 

0.1 

-0.07 

4.9 

4.8 

0.2 

0.2 

0.14 


Pre-academic skills 












Language and literacy skills 

33.1 

30.6 

2.4 

2.7 

0.25 

36.3 

34.0 

2.3 

2.1 

0.29 


Math knowledge 

24.6 

23.3 

1.3 

2.3 

0.17 

27.6 

26.0 

1.6 

2.4 

0.24 


Sample size 












Observations of students 

54 

51 




40 

31 





Teacher reports on students 

124 

126 




89 

77 





Classrooms 0 

26 

25 




26 

25 






SOURCES: Based on MDRC calculations of Individualized Classroom Assessment Scoring System (inCLASS) observations in April-May 2008 and on 
responses to teacher survey. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. The H-statistic test was used to test for 
statistically significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as follows: fff = 1 
percent; f f = 5 percent; f = 10 percent. 

Regression-adjusted means control for random assignment status and blocking, baseline Classroom Assessment Scoring System (CLASS) measures, 
and baseline child characteristics. Teacher -reported outcomes control for the student's baseline score on a given measure, when available. These include 
baseline measures for the Cooper-Farran Behavioral Rating Scales, the Behavior Problems Index (BPI), and the Positive Behavior Scale. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the control group. 

a For each observed inCLASS dimension, observers rated children on a scale from 1 to 7, with 1 representing "low" and 7 representing "high." 

b "Intemalizing problems" and "Externalizing problems" refer to the internalizing and externalizing subscales of the BPI. "Teacher-student conflict" 
and "Teacher-student closeness" refer to the conflict and closeness subscales of the Student-Teacher Relationship Scale. "Social competence" refers to 
the social competence subscale of the Positive Behavior Scale. "Work-related skills" refers to the work-related skills subscale of the Cooper-Farran 
Behavioral Rating Scales. "Language and literacy skills" and "Math knowledge" refer to the language and literacy and math knowledge subscales of the 
Academic Rating Scale. 

c For teacher-reported outcomes, the sample size is 23 for the control group classrooms. 
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Newark Program Impacts on Observed and Teacher Ratings of Child Outcomes, 
by Child's Level of Behavior Problems 





Low 





High 



Subgroup 

Outcome 

Mean Score 
Program Control 
Group Group 

Difference 

(Impact) 

Standard 

Error 

Effect 

Size 

Mean Score 
Program Control 
Group Group 

Difference 

(Impact) 

Standard 

Error 

Effect 

Size 

Impact 

Difference 

(Sig.) 

Observations 11 

Problem behavior 

Teacher conflict 

1.3 

1.3 

0.0 

0.1 

0.04 

1.4 

1.6 

-0.2 

0.1 

-0.40 


Peer conflict 

1.5 

1.5 

0.0 

0.2 

-0.06 

1.5 

1.8 

-0.3 ** 

0.1 

-0.51 


Positive social behavior 

Teacher communication 

2.1 

2.5 

-0.4 * 

0.2 

-0.47 

2.3 

2.3 

0.0 

0.2 

-0.01 


Teacher positive engagement 

3.1 

3.5 

-0.4 

0.2 

-0.46 

3.2 

3.3 

-0.1 

0.3 

-0.13 


Peer communication 

2.6 

2.6 

0.0 

0.3 

-0.05 

2.5 

2.6 

-0.1 

0.2 

-0.18 


Peer sociability 

3.5 

3.7 

-0.2 

0.2 

-0.24 

3.4 

3.6 

-0.2 

0.2 

-0.23 


Peer assertiveness 

2.0 

2.3 

-0.3 

0.4 

-0.29 

2.2 

2.2 

-0.1 

0.2 

-0.09 


Approach to learning 

Task engagement 

4.9 

4.8 

0.2 

0.2 

0.19 

4.8 

4.6 

0.2 

0.2 

0.27 


Task self-reliance 

2.9 

3.1 

-0.1 

0.4 

-0.09 

3.1 

3.2 

-0.1 

0.3 

-0.08 


Task behavior control 

5.6 

5.2 

0.3 

0.3 

0.36 

5.1 

4.8 

0.3 

0.2 

0.35 


Teacher renorts b 

Problem behavior 

Internalizing problems 

1.6 

1.2 

0.4 

0.5 

0.18 

4.0 

2.6 

1.4 

1.1 

0.39 


Externalizing problems 

1.7 

1.7 

0.0 

0.6 

-0.01 

6.9 

5.2 

1.7 

1.3 

0.28 


Teacher-student conflict 

9.6 

10.0 

-0.4 

0.9 

-0.10 

15.3 

13.5 

1.9 

1.4 

0.26 


Positive social behavior 

Social competence 

4.3 

4.2 

0.1 

0.2 

0.09 

3.7 

3.8 

0.0 

0.1 

-0.06 


Teacher-student closeness 

35.8 

35.8 

0.0 

1.1 

0.01 

33.5 

35.9 

-2.4 * 

1.2 

-0.39 
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Low 





High 



Subgroup 


Mean Score 




Mean Score 




Impact 


Program 

Control 

Difference 

Standard 

Effect 

Program 

Control 

Difference 

Standard 

Effect 

Difference 

Outcome 

Group 

Group 

(Impact) 

Error 

Size 

Group 

Group 

(Impact) 

Error 

Size 

(Sig.) 

Teacher reports (continued) 1 ’ 

Approach to learning 












Work-related skills 

5.1 

5.2 

-0.1 

0.2 

-0.09 

4.5 

4.6 

-0.1 

0.2 

-0.08 


Preacademic skills 












Language and literacy skills 

36.1 

34.9 

1.2 

1.7 

0.13 

32.8 

32.5 

0.3 

2.0 

0.03 


Math knowledge 

26.7 

27.3 

-0.6 

1.8 

-0.08 

23.8 

25.3 

-1.5 

1.9 

-0.21 


Sample size 












Observations of students 

50 

40 




58 

38 





Teacher reports on students 

125 

94 




128 

100 





Classrooms c 

26 

25 




26 

25 






SOURCES: Based on MDRC calculations of classroom observations in April-May 2008 and responses to teacher survey. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. The H-statistic test was used to test for statistically 
significant differences in impact estimates across different subgroups. Statistical significance levels are indicated as follows: ff f = 1 percent; f f = 5 percent; 
f = 10 percent. The effect size equals the impact divided by the standard deviation of the outcome measure for the control group. Regression-adjusted means 
control for random assignment status and blocking, baseline Classroom Assessment Scoring System (CLASS) measures, and baseline child characteristics. 

This subgroup was created by calculating a median split of baseline teacher-reported BPI scores across the entire sample. BPI scores higher than a 4 fall 
into the "high" category. Baseline BPI scores were not available for all children. 

Tor each observed Individualized Classroom Assessment Scoring System (inCLASS) dimension, observers rated children on a scale from 1 to 7, with 1 
representing "low" and 7 representing "high." 

b Teacher-reported outcomes control for the student's baseline score on a given measure, when available. These include baseline measures for the Cooper- 
Farran Behavioral Rating Scales, the Behavior Problems Index (BPI), and the Positive Behavior Scale. 

"Internalizing problems" and "Externalizing problems" refer to the internalizing and externalizing subscales of the BPI. "Teacher-student conflict" and 
"Teacher-student closeness" refer to the Student-Teacher Relationship Scale conflict and closeness subscales. "Attention problems" refers to the Caregiver- 
Teacher Report Form attention problems subscale. "Compliance with teachers' directives" and "Social competence" refer to the Positive Behavior Scale 
compliance and social competence subscales. "Work-related skills" refers to the Cooper-Farran Behavioral Rating Scales work-related skills subscale. 
"Language and literacy skills" and "Math knowledge" refer to the subscales of the Academic Rating Scale. 

Tor teacher-reported outcomes, the sample size is 23 for the control group classrooms. 
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Newark Program Impacts on Teacher-Reported Child Outcomes, Kindergarten Year 


Mean Score 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Standard 

Error 

Effect 

Size 

Problem behavior 2 

Internalizing problems 

3.2 

2.4 

0.8 * 

0.4 

0.2 

Externalizing problems 

4.7 

3.8 

1.0 * 

0.6 

0.2 

Teacher-student conflict 

12.3 

11.5 

0.8 

0.7 

0.1 

Attention problems 

4.4 

4.3 

0.1 

0.5 

0.0 

Positive social behavior b 

Compliance with teachers' directives 

4.0 

4.0 

-0.1 

0.1 

-0.1 

Social competence 

4.0 

4.0 

0.0 

0.1 

0.0 

Teacher-student closeness 

33.1 

33.9 

-0.8 

0.6 

-0.1 

Approach to learning 

Work-related skills 0 

4.8 

4.9 

0.0 

0.1 

0.0 

Academic skills 11 

Language and literacy skills 

3.8 

3.7 

0.0 

0.1 

0.0 

Math knowledge 

3.8 

3.8 

0.0 

0.1 

0.0 

General knowledge 

3.8 

3.9 

0.0 

0.1 

0.0 

Sample size 

259 

230 





SOURCE: Based on MDRC calculations from responses to teacher survey. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Regression-adjusted means control for random assignment status and blocking, baseline Classroom Assessment 
Scoring System (CLASS) measures, and baseline child characteristics. 

Controls for the child's baseline score on a given measure are also included, when available. These include 
baseline measures for the Cooper-Farran Behavioral Rating Scales, the Behavior Problems Index (BPI), and the 
Positive Behavior Scale. 

The effect size equals the impact divided by the standard deviation of the outcome measure for the control 
group. 

“"Internalizing problems" and "Externalizing problems" refer to the internalizing and externalizing subscales of 
the BPI. "Teacher-student conflict" refers to the Student-Teacher Relationship Scale (STRS) conflict subscale. 
"Attention problems" refers to the Caregiver-Teacher Report Form attention problems subscale. 

b "Compliance with teachers' directives" and "Social competence" refer to the Positive Behavior Scale 
compliance and social competence subscales. "Teacher-student closeness" refers to the closeness of the STRS. 
c "Work-related skills" refers to the Cooper-Farran Behavioral Rating Scales work-related skills subscale. 
d "Language and literacy skills," "Math knowledge," and "General knowledge" refer to the subscales of the 
Academic Rating Scale. 
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Chicago Program Impacts on Child Outcomes, Kindergarten Year, by Level of School Quality 


Schools with Lower Test Scores Schools with Higher Test Scores Subgroup 


Outcome 

Program Control 
Group Group Difference 
Mean Mean (Impact) 

Standard 

Error 

Effect 

Size 

Program 

Group 

Mean 

Control 

Group Difference 
Mean (Impact) 

Standard 

Error 

Effect 

Size 

Impact 

Difference 

(Sig.) 

Early literacy standardized score 3 

0.0 

0.1 

-0.2 

0.21 

-0.23 

-0.1 

-0.2 

0.1 

0.30 

0.09 


Recommended for intensive 

instructional support (%) 

13.6 

17.7 

-4.2 

7.93 

-0.15 

15.5 

7.6 

7.9 

13.13 

0.24 


Poor attendance 13 (%) 

19.9 

19.2 

0.8 

10.47 

0.02 

17.9 

8.5 

9.5 

13.08 

0.29 


Disability (%) 

Social-emotional disorder 3 

-0.9 

2.2 

-3.1 

2.80 

-0.19 

— 

— 

— 

— 

— 


Social-emotional or learning disability 

1.8 

6.5 

-4.7 

5.66 

-0.18 

7.8 

-5.9 

13.7 * 

6.17 

0.66 

ft 

Sample size 

72 

71 




48 

45 






SOURCE: Based on MDRC calculations from Chicago Public School student records. 


NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. The effect size equals the impact divided by the 
standard deviation of the outcome measure for the control group. The El-statistic test was used to test for statistically significant differences in impact estimates 
across different subgroups. Statistical significance levels are indicated as follows: fff = 1 percent; f f = 5 percent; f = 10 percent. 

Regression-adjusted means control for random assignment status, matched pairs, and baseline child characteristics. Schools with higher test scores have 60 
percent or more of the school's students meeting or exceeding Illinois Standard Achievement Test (ISAT) reading expectations and 70 percent or more of the 
school's students meeting or exceeding ISAT math expectations. 

a Scores from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) are presented as standardized scores with a mean of 0 and a standard deviation 
of 1. 

b "Poor attendance" means that the child was absent 10 percent of the time or more during the school year (measured in days). 

information for social-emotional disorders for schools with higher test scores are represented as long dashes (— ) because there were no children with social- 
emotional disorders in those classrooms. 
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Estimated FOL Program Costs, Newark and Chicago 



Total Cost 

Cost per 

Classroom 

Cost per 

Student 

Component 

Newark 

Chicago 

Newark 

Chicago 

Newark 

Chicago 

Teacher trainings ($) 

132,560 

109,956 

5,098 

5,498 

340 

275 

Trainer fees and travel costs 

38,291 

38,012 

1,473 

1,901 

98 

95 

Lead teacher payments 

35,832 

25,610 

1,378 

1,280 

92 

64 

Assistant teacher payments 

17,374 

11,925 

668 

596 

45 

30 

Clinical classroom consultant payments 

12,101 

7,488 

465 

374 

31 

19 

Child care payments 

3,044 

7,745 

117 

387 

8 

19 

Incredible Years book 

2,068 

1,516 

80 

76 

5 

4 

Location rental and catering 

20,461 

17,007 

787 

850 

52 

43 

Other expenses 3 

3,390 

653 

130 

33 

9 

2 

Clinical classroom consultations ($) 

507,638 

534,429 

19,525 

26,721 

1,302 

1,336 

Clinical classroom consultant salaries b 

307,625 

258,593 

11,832 

12,930 

789 

646 

Administration and supervision 

88,638 

152,959 

3,409 

7,648 

227 

382 

Unemployment b 

9,015 

0 

347 

— 

23 

— 

Stress management workshop 6 

4,531 

998 

174 

50 

12 

2 

Clinical classroom consultant orientation 

1,772 

1,643 

68 

82 

5 

4 

Other expenses* 3 

96,057 

120,236 

3,694 

6,012 

246 

301 

Support for the intervention ($) 

Technical assistance d 

58,501 

53,290 

2,250 

2,665 

150 

133 

Total expenditures for all components 6 ($) 

698,699 

697,675 

26,873 

34,884 

1,792 

1,744 


(continued) 





Appendix Table D.l (continued) 


SOURCES: MDRC calculations based on fiscal and budget data. 

NOTES: Costs are shown for 26 classrooms in Newark with an average of 15 students per classroom, and 20 classrooms in Chicago 
with an average of 20 students per classroom. All costs are shown in 20 1 0 dollars. 
a Other expenses cover additional supplies purchased for training sessions. 

b The agency used in Newark was Family Connections; the agency used in Chicago was the Institute for Positive Child and Family 
Development. The numbers for the Institute for Positive Child and Family Development come from the initial budget provided to 
MDRC. The Institute did not provide a final accurate list of expenditures. 

includes catering, payment for teachers, an incentive for teachers to attend (Newark only), and additional supplies. Workshops 
were held at individual preschool centers. 

d Technical assistance was provided by MDRC. 

e Total expenditures include all teacher training expenditures, classroom consultant expenditures, and MDRC management. 
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